Posts

Are there good ways to find expert reviews of popular science books? 2020-06-09T14:54:23.102Z · score: 25 (6 votes)
Three characteristics: impermanence 2020-06-05T07:48:02.098Z · score: 51 (19 votes)
On the construction of the self 2020-05-29T13:04:30.071Z · score: 46 (17 votes)
From self to craving (three characteristics series) 2020-05-22T12:16:42.697Z · score: 37 (16 votes)
Craving, suffering, and predictive processing (three characteristics series) 2020-05-15T13:21:50.666Z · score: 52 (21 votes)
A non-mystical explanation of "no-self" (three characteristics series) 2020-05-08T10:37:06.591Z · score: 67 (28 votes)
A non-mystical explanation of insight meditation and the three characteristics of existence: introduction and preamble 2020-05-05T19:09:44.484Z · score: 91 (31 votes)
Stanford Encyclopedia of Philosophy on AI ethics and superintelligence 2020-05-02T07:35:36.997Z · score: 42 (18 votes)
Healing vs. exercise analogies for emotional work 2020-01-27T19:10:01.477Z · score: 41 (21 votes)
The two-layer model of human values, and problems with synthesizing preferences 2020-01-24T15:17:33.638Z · score: 69 (22 votes)
Under what circumstances is "don't look at existing research" good advice? 2019-12-13T13:59:52.889Z · score: 73 (22 votes)
A mechanistic model of meditation 2019-11-06T21:37:03.819Z · score: 107 (36 votes)
On Internal Family Systems and multi-agent minds: a reply to PJ Eby 2019-10-29T14:56:19.590Z · score: 38 (16 votes)
Book summary: Unlocking the Emotional Brain 2019-10-08T19:11:23.578Z · score: 187 (80 votes)
Against "System 1" and "System 2" (subagent sequence) 2019-09-25T08:39:08.011Z · score: 92 (29 votes)
Subagents, trauma and rationality 2019-08-14T13:14:46.838Z · score: 72 (39 votes)
Subagents, neural Turing machines, thought selection, and blindspots 2019-08-06T21:15:24.400Z · score: 67 (25 votes)
On pointless waiting 2019-06-10T08:58:56.018Z · score: 43 (22 votes)
Integrating disagreeing subagents 2019-05-14T14:06:55.632Z · score: 96 (30 votes)
Subagents, akrasia, and coherence in humans 2019-03-25T14:24:18.095Z · score: 101 (33 votes)
Subagents, introspective awareness, and blending 2019-03-02T12:53:47.282Z · score: 67 (26 votes)
Building up to an Internal Family Systems model 2019-01-26T12:25:11.162Z · score: 167 (67 votes)
Book Summary: Consciousness and the Brain 2019-01-16T14:43:59.202Z · score: 117 (45 votes)
Sequence introduction: non-agent and multiagent models of mind 2019-01-07T14:12:30.297Z · score: 94 (41 votes)
18-month follow-up on my self-concept work 2018-12-18T17:40:03.941Z · score: 58 (17 votes)
Tentatively considering emotional stories (IFS and “getting into Self”) 2018-11-30T07:40:02.710Z · score: 39 (11 votes)
Incorrect hypotheses point to correct observations 2018-11-20T21:10:02.867Z · score: 81 (34 votes)
Mark Eichenlaub: How to develop scientific intuition 2018-10-23T13:30:03.252Z · score: 81 (31 votes)
On insecurity as a friend 2018-10-09T18:30:03.782Z · score: 38 (20 votes)
Tradition is Smarter Than You Are 2018-09-19T17:54:32.519Z · score: 68 (24 votes)
nostalgebraist - bayes: a kinda-sorta masterpost 2018-09-04T11:08:44.170Z · score: 24 (8 votes)
New paper: Long-Term Trajectories of Human Civilization 2018-08-12T09:10:01.962Z · score: 34 (16 votes)
Finland Museum Tour 1/??: Tampere Art Museum 2018-08-03T15:00:05.749Z · score: 20 (6 votes)
What are your plans for the evening of the apocalypse? 2018-08-02T08:30:05.174Z · score: 24 (11 votes)
Anti-tribalism and positive mental health as high-value cause areas 2018-08-02T08:30:04.961Z · score: 26 (10 votes)
Fixing science via a basic income 2018-08-02T08:30:04.380Z · score: 30 (14 votes)
Study on what makes people approve or condemn mind upload technology; references LW 2018-07-10T17:14:51.753Z · score: 21 (11 votes)
Shaping economic incentives for collaborative AGI 2018-06-29T16:26:32.213Z · score: 47 (13 votes)
Against accusing people of motte and bailey 2018-06-03T21:31:24.591Z · score: 84 (28 votes)
AGI Safety Literature Review (Everitt, Lea & Hutter 2018) 2018-05-04T08:56:26.719Z · score: 37 (10 votes)
Kaj's shortform feed 2018-03-31T13:02:47.793Z · score: 13 (3 votes)
Helsinki SSC March meetup 2018-03-26T19:27:17.850Z · score: 12 (2 votes)
Is the Star Trek Federation really incapable of building AI? 2018-03-18T10:30:03.320Z · score: 30 (9 votes)
My attempt to explain Looking, insight meditation, and enlightenment in non-mysterious terms 2018-03-08T07:37:54.532Z · score: 297 (120 votes)
Some conceptual highlights from “Disjunctive Scenarios of Catastrophic AI Risk” 2018-02-12T12:30:04.401Z · score: 68 (20 votes)
On not getting swept away by mental content 2018-01-25T20:30:03.750Z · score: 25 (9 votes)
Papers for 2017 2018-01-04T13:30:01.406Z · score: 32 (8 votes)
Paper: Superintelligence as a Cause or Cure for Risks of Astronomical Suffering 2018-01-03T14:39:18.024Z · score: 1 (1 votes)
Paper: Superintelligence as a Cause or Cure for Risks of Astronomical Suffering 2018-01-03T13:57:55.979Z · score: 16 (6 votes)
Fixing science via a basic income 2017-12-08T14:20:04.623Z · score: 38 (11 votes)

Comments

Comment by kaj_sotala on Noise on the Channel · 2020-07-02T09:45:56.034Z · score: 10 (2 votes) · LW · GW
Literal unironic object-level question: why do so many people think this is a good social setting? Maybe the noise serves an important social function I'm not seeing?

A little while back, I was reading this article which talks, among other things, about how COVID restrictions will change the atmosphere in restaurants. I thought that the writer would say something like "having fewer customers at a time will make it harder for restaurants to profit, but will contribute to a less crowded and pleasantly quiet atmosphere for those customers that can get seats". Instead I got this, which sounds like the author actively enjoys the noise:

Empty space is bad enough for downtown restaurants, where thin margins require filling every square inch with paying customers. But at a deeper level, these adaptations will create a whole new ambience, making restaurants more awkward, more expensive, and less fun. One of the joys of getting a drink in a crowded space is the soundtrack of a hundred strangers’ conversations humming underneath the intimacy of a private exchange. Social-distance dining prohibits the thrum of a full house.

Thinking about it, some people even use artificial crowd noises to help them work, so that sound may feel actively enjoyable to many.

Comment by kaj_sotala on GPT-3 Fiction Samples · 2020-07-02T07:56:44.466Z · score: 2 (1 votes) · LW · GW

I'm not sure how exactly reasoning should be defined and whether that part really requires reasoning or not. But if it's just very advanced and incredible recognition and mimicry abilities, it still shifts my impression of what can be achieved using just advanced and incredible recognition and mimicry abilities. I would previously have assumed that you need something like reasoning for it, but if you don't, then maybe the capacity for reasoning is slightly less important than I had thought.

Comment by kaj_sotala on GPT-3 Fiction Samples · 2020-07-01T13:12:05.343Z · score: 4 (2 votes) · LW · GW

My largest update came from the bit where it figured out that it was expected to produce Harry Potter parodies in different styles. Previously GPT had felt cool, but basically a very advanced version of a Markov chain. But the HP thing felt like it would have required some kind of reasoning.

Comment by kaj_sotala on What's Your Cognitive Algorithm? · 2020-07-01T12:52:59.337Z · score: 4 (2 votes) · LW · GW
Is your claim "this is insufficient – you still need working memory and the ability to model scenarios, and currently we don't know how to do that, and there are good reasons to think that throwing lots of data and better reward structures at our existing algorithms won't be enough to cause this to develop automatically via Neural Net Magic?"

So at this point I'm pretty uncertain of what neural nets can or can not learn to do. But at least I am confident in saying that GPT isn't going to learn the kinds of abilities that would be required for actually fighting fires, as it is trained and tested on a fundamentally static task, as opposed to one that requires adapting your behavior to a situation as it develops. For evaluating at progress on those, projects like AlphaStar look like more relevant candidates.

I don't feel confident in saying whether some combination of existing algorithms and training methods could produce a system that approached the human level on dynamic tasks. Most people seem to agree that we haven't gotten neural nets to learn to do good causal reasoning yet, so my understanding of the expert consensus is that current techniques seem inadequate... but then the previous expert consensus would probably also have judged neural nets to be inadequate for doing many of the tasks that they've now mastered.

Comment by kaj_sotala on How ought I spend time? · 2020-06-30T17:12:30.075Z · score: 3 (2 votes) · LW · GW

Whether they feel motivating/exciting/interesting enough for me to actually carry them out. Things like "will this have an impact on anything" do get factored into that, e.g. completely pointless activities often feel less interesting. But my conscious mind only catches glimpses of that calculation, so I wouldn't call it "reasoning".

Comment by kaj_sotala on Optimized Propaganda with Bayesian Networks: Comment on "Articulating Lay Theories Through Graphical Models" · 2020-06-30T17:04:34.408Z · score: 4 (2 votes) · LW · GW
He was bitter. He was angry. He wouldn’t look at her. And, she could recognize, what he was saying didn’t make sense. She recognized, what he was saying didn’t fit with what he had just said. ... What Crowley was saying didn’t make sense, not on the surface. The individual pieces of what he said were incoherent, which, she knew, meant that there must be some other layer, some deeper layer, where they did make sense.
Comment by kaj_sotala on Don't Make Your Problems Hide · 2020-06-29T06:11:44.759Z · score: 8 (4 votes) · LW · GW
There are other approaches too. Many people believe in using meditation to better integrate their thoughts and feelings and desires, for instance.

Worth noting that at least some styles of meditation can also make the problems hide. As with CFAR/CBT techniques, some of the issues get dissected and resolved, but others may just get reburied deeper.

Comment by kaj_sotala on GPT-3 Fiction Samples · 2020-06-25T21:41:01.721Z · score: 8 (5 votes) · LW · GW

Okay, my intuitions for AI timelines just shifted to put considerably more probability on the short end.

Comment by kaj_sotala on Spoiler-Free Review: Monster Train · 2020-06-25T07:45:29.023Z · score: 2 (1 votes) · LW · GW
Note that I still say Class here rather than Class Pair. In my experience, you lean into one or the other theme, and use only a little from the other. There are good units and a few good spells everywhere, but you can’t mix multiple card-intensive themes if you want to win.

This feels often true, but I just noticed that Hellhorned/Remnant can work really well together. Spam imps for their effects, let them die / kill them off with Inferno or Imp-portant Work, then reform them and repeat.

Had a particularly enjoyable Covenant 9 run ( monstertrain://runresult/21069f33-c51b-4b3f-b12a-a22a6c4c7c52 ) where I also lucked into having Flicker's Liquor and it was just insane, all of my heavy hitters on the top floor and the lower floors swarming with imps and dregs. I was mostly playing them for their effects and didn't even care if they killed enemies (because the top floor would have taken care of anyone anyway), but they had been reformed so many times that they actually dealt significant damage on their own. Infernos helped too, and many enemy waves never even made it to the top. Seraph was consuming spells but my Impish Scholars meant that that didn't really even matter: often the consumed spell would already be back in my hand before the turn was over. At best I was playing something like four copies of Onehorn's Tome (with a total of two copies in my deck) a turn, and the battle was over before it ever even got to the final wave.

Comment by kaj_sotala on Neural Basis for Global Workspace Theory · 2020-06-23T13:58:40.793Z · score: 4 (2 votes) · LW · GW

Cool, done. :)

There seems to me to be a conceptual difference between the kinds of actions that change the contents of consciousness, and the kinds of actions which accumulate evidence over many items in consciousness (such as iterative memories of snacks). Zylberberg et al. talk about a “winner-take-all race” to trigger a production rule, which to me implies that the evidence accumulated in favor of each production rule is cleared each time that the contents of consciousness is changed. This is seemingly incompatible with accumulating evidence over many consciousness-moments, so postulating a two-level distinction between accumulators seems like a straightforward way of resolving the issue.
[EDIT: Hazard suggests that the two-level split is implemented by the basal ganglia carrying out evidence accumulation across changes in conscious content.]
Comment by kaj_sotala on SlateStarCodex deleted because NYT wants to dox Scott · 2020-06-23T13:50:46.485Z · score: 23 (11 votes) · LW · GW

Especially since Scott explicitly asked people to be nice, and has written whole essays about not using those kinds of tactics.

Comment by kaj_sotala on The "hard" problem of consciousness is the least interesting problem of consciousness · 2020-06-23T11:19:43.272Z · score: 2 (1 votes) · LW · GW
The Hard Problem is basically "what part of the equation for wavefunction of the universe says that we are not zombies". The answer of panpsychism is "the part where we say that it is real".

I don't think I understand. I would say that the Hard Problem is more "why and how do we have subjective experience, rather than experiencing nothing". If you say that "everything has it", that doesn't seem to answer the question - okay, everything is conscious, but why and how is everything conscious?

Comment by kaj_sotala on SlateStarCodex deleted because NYT wants to dox Scott · 2020-06-23T09:59:09.354Z · score: 39 (20 votes) · LW · GW

I feel like the more places report on this, the higher the probability that at least one of them will publish Scott's real name.

Comment by kaj_sotala on Neural Basis for Global Workspace Theory · 2020-06-22T18:28:04.864Z · score: 9 (2 votes) · LW · GW

Something being on the GNW can boost evidence accumulation at the basal ganglia, which is maintained across changes in the contents of GNW.

Nice! That certainly clarifies things. :) Mind if I edit my article to include a reference to your post?

Comment by kaj_sotala on Image GPT · 2020-06-21T10:59:11.243Z · score: 4 (2 votes) · LW · GW

I don't know whether reasoning is a roadblock or not, but I discuss some ways in which GPT doesn't have it in this comment.

Comment by kaj_sotala on What's Your Cognitive Algorithm? · 2020-06-21T09:55:14.131Z · score: 16 (4 votes) · LW · GW

For my paper "How Feasible is the Rapid Development of Artificial Superintelligence?", I looked at some of the existing literature on human expertise to develop a model of exactly what it is that human intelligence consists of.

As a very rough distinction, somewhat analogous to Type 1 and Type 2 reasoning, we can divide human expertise into two components: pattern recognition and mental simulation. An excerpt:

There exists a preliminary understanding, if not of the details of human decision-making, then at least the general outline. A picture that emerges from this research is that expertise is about developing the correct mental representations (Klein 1999, Ericsson and Pool 2016).

A mental representation is a very general concept, roughly corresponding to any mental structure forming the content of something that the brain is thinking about (Ericsson and Pool 2016).

Domain-specific mental representations are important because they allow experts to know what something means; know what to expect; know what good performance should feel like; know how to achieve the good performance; know the right goals for a given situation; know the steps necessary for achieving those goals; mentally simulate how something might happen; learn more detailed mental representations for improving their skills (Klein 1999, Ericsson and Pool 2016).

Although good decision-making is often thought of as a careful deliberation of all the possible options, such a type of thinking tends to be typical of novices (Klein 1999). A novice will have to try to carefully reason their way through to an answer, and will often do poorly regardless, because they do not know what things are relevant to take into account and which ones are not. An expert does not need to—they are experienced enough to instantly know what to do.
A specific model of expertise is the recognition-primed decision-making model (Klein 1999). First, a decision-maker sees some situation, such as a fire for a firefighter or a design problem for an architect. The situation may then be recognized as familiar, such as a typical garage fire. Recognizing a familiar situation means understanding what goals make sense and what should be focused on, which cues to pay attention to, what to expect next and when a violation of expectations shows that something is amiss, and knowing what the typical ways of responding are. Ideally, the expert will instantly know what to do.

The expectations arising from mental representations also give rise to intuition. As one example, Klein (1999) describes the case of a firefighter lieutenant responding to a kitchen fire in an ordinary one-story residential house. The lieutenant’s crew sprayed water on the fire, but contrary to expectations, the water seemed to have little impact. Something about the situation seemed wrong to the lieutenant, who ordered his crew out of the house. As soon as they had left the house, the floor where they had been standing collapsed. If the firefighters had not pulled out, they would have fallen down to the fire raging in the basement. The lieutenant, not knowing what had caused him to give the order to withdraw, initially attributed the decision to some form of extra-sensory perception.

In a later interview, the lieutenant explained that he did not suspect that the building had a basement, nor that the seat of the fire was under the floor that he and his crew were standing on. However, several of his expectations of a typical kitchen fire were violated by the situation. The lieutenant was wondering why the fire did not react to water as expected, the room was much hotter than he would have expected out of a small kitchen fire, and while a heat that hot should have made a great deal of noise, it was very quiet. The mismatch between the expected pattern and the actual situation led to an intuitive feeling of not knowing what was going on, leading to the decision to regroup. This is intuition: an automatic comparison of the situation against existing mental representations of similar situations, guiding decision-making in ways whose reasons are not always consciously available.

In an unfamiliar situation, the expert may need to construct a mental simulation of what is going on, how things might have developed to this point, and what effect different actions would have. Had the floor mentioned in the previous example not collapsed, given time the firefighter lieutenant might have been able to put the pieces together and construct a narrative of a fire starting from the basement to explain the discrepancies. For a future-oriented example, a firefighter thinking about how to rescue someone from a difficult spot might mentally simulate where different rescue harnesses might be attached on the person, and whether that would exert dangerous amounts of force on them.

Mental representations are necessary for a good simulation, as they let the expert know what things to take into account, what things could plausibly be tried, and what effects they would have. In the example, the firefighter’s knowledge allows him to predict that specific ways of attaching the rescue harness would have dangerous consequences, while others are safe.

Similar to the firefighter's intuition, GPT-2 has the ability to make predictions about what's most likely to "happen next". But there are also several differences.

Most notably, GPT-2's only goal is just that: predict what's going to happen next. This is a much more limited task than the one faced by (for example) a human firefighter, who needs to not just predict how a fire might proceed, but also how to best respond to it and which actions to take.

Let's take a concrete example and look at it in more detail; from Klein 1999:

The initial report is of flames in the basement of a four-story apartment building: a one-alarm fire. The [firefighter] commander arrives quickly and does not see anything. There are no signs of smoke anywhere. He finds the door to the basement, around the side of the building, enters, and sees flames spreading up the laundry chute. That's simple: a vertical fire that will spread straight up. Since there are no external signs of smoke, it must just be starting.

The way to fight a vertical fire is to get above it and spray water down, so he sends one crew up to the first floor and another to the second floor. Both report that the fire has gotten past them. The commander goes outside and walks around to the front of the building. Now he can see smoke coming out from under the eaves of the roof.

It is obvious what has happened: the fire has gone straight up to the fourth floor, has hit the ceiling there, and is pushing smoke down the hall. Since there was no smoke when he arrived just a minute earlier, this must have just happened. It is obvious to him how to proceed now that the chance to put out the fire quickly is gone. He needs to switch to search and rescue, to get everyone out of the building, and he calls in a second alarm. The side staircase near the laundry chute had been the focus of activity before. Now the attention shifts to the front stairway as the evacuation route.

And this picture shows the general algorithm for recognition-primed decision-making. Breaking down the story, we might say that something like the following happened:

1. The commander sees no smoke outside, then flames spreading up the laundry chute. These are the cues that allow him to recognize this as a vertical fire that is just starting.

2. The commander's mental representation of vertical fires includes plausible goals, expectancies of what is going to happen, and actions that could further the goals. A plausible goal for this situation: put the fire out quickly, before it has a chance to spread. An action that would further it: send people to spray water on the fire from above. A rapid mental simulation suggests that this should work, so he gives the order.

3. The crews report that the fire has gotten past them. This violates the expectancy that the fire should be in the basement only; to diagnose this anomaly, the commander goes outside to gather more data. When he sees the smoke coming up from the roof, this allows him to construct a story of what has happened.

4. The situation is now different, calling up a new mental representation: that of a fire that has spread from the basement to the top floor. Plausible goals in this situation: get everyone out of the building. Actions to take here: call in reinforcements, get people to the front stairway to carry out an evacuation.

As at least one major difference, GPT-2 never does the thing where it expects that something will happen, and then takes actions to re-evaluate the situation if the prediction goes wrong. If it predicts "the word after 'maximize' is going to be 'paperclip'" with 90% confidence, finding out that it's actually followed by 'human values' doesn't cause it to...

Actually, I don't need to complete that sentence, because "seeing that it was mistaken" isn't actually a thing that happens to GPT-2 in the first place. It does get feedback to its predictions during its training phase, but once it has been trained, it will never again compare its prediction with the actual result. You just give it a prompt and then it tries to predict the rest, that's it. If you give it one prompt, have it predict the rest of it, and then give it a revised prompt with the correct completion, it has no idea that you are doing this. It just sees one prompt and then another. This makes it incapable of noticing that its expectations are violated, gathering more information in return, and then constructing a story of what happened and what kind of a situation it's actually in.

You could probably apply it to something like "predict what a human firefighter would do in this situation" (imitation learning), but as anyone can verify by playing AI Dungeon (which now uses GPT-3 not GPT-2), its predictions get very nonsensical very quickly. It doesn't really do the kind of causal reasoning that would involve mental simulations to produce novel responses, e.g. the following example from Klein:

A [firefighter] lieutenant is called out to rescue a woman who either fell or jumped off a highway way overpass. She is drunk or on drugs and is probably trying to kill herself. Instead of falling to her death, she lands on the metal supports of a highway sign and is dangling there when the rescue team arrives.

The lieutenant recognizes the danger of the situation. The woman is semiconscious and lying bent over one of the metal struts. At any moment, she could fall to her death on the pavement below. If he orders any of his team out to help her, they will be endangered because there is no way to get a good brace against the struts, so he issues an order not to climb out to secure her.

Two of his crew ignore his order and climb out anyway. One holds onto her shoulders and the other to her legs.

A hook-and-ladder truck arrives. The lieutenant doesn't need their help in making the rescue, so tells them to drive down to the highway below and block traffic in case the woman does fall. He does not want to chance that the young woman will fall on a moving car.

Now the question is how to pull the woman to safety.

First, the lieutenant considers using a rescue harness, the standard way of raising victims. It snaps onto a person's shoulders and thighs. In imagining its use, he realizes that it requires the person to be in a sitting position or face up. He thinks about how they would shift her to sit up and realizes that she might slide off the support.

Second, he considers attaching the rescue harness from the back. However, he imagines that by lifting the woman, they would create a large pressure on her back, almost bending her double. He does not want to risk hurting her.

Third, the lieutenant considers using a rescue strap-another way to secure victims, but making use of a strap rather than a snap-on harness. However, it creates the same problems as the rescue harness, requiring that she be sitting up or that it be attached from behind. He rejects this too.

Now he comes up with a novel idea: using a ladder belt-a strong belt that firefighters buckle on over their coats when they climb up ladders to rescue people. ple. When they get to the top, they can snap an attachment on the belt to the top rung of the ladder. If they lose their footing during the rescue, they are still attached to the ladder so they won't plunge to their death.

The lieutenant's idea is to get a ladder belt, slide it under the woman, buckle it from behind (it needs only one buckle), tie a rope to the snap, and lift her up to the overpass. He thinks it through again and likes the idea, so he orders one of his crew to fetch the ladder belt and rope, and they tie it onto her.

In the meantime, the hook-and-ladder truck has moved to the highway below the overpass, and the truck's crew members raise the ladder. The firefighter on the platform at the top of the ladder is directly under the woman shouting, "I've got her. I've got her." The lieutenant ignores him and orders his men to lift her up.

At this time, he makes an unwanted discovery: ladder belts are built for sturdy firefighters, to be worn over their coats. This is a slender woman wearing a thin sweater. In addition, she is essentially unconscious. When they lift her up, they realize the problem. As the lieutenant put it, "She slithered through the belt like a slippery strand of spaghetti."

Fortunately, the hook-and-ladder man is right below her. He catches her and makes the rescue. There is a happy ending.

Now the lieutenant and his crew go back to their station to figure out what had gone wrong. They try the rescue harness and find that the lieutenant's instincts were right: neither is usable.

Eventually they discover how they should have made the rescue. They should have used the rope they had tied to the ladder belt. They could have tied it to the woman and lifted her up. With all the technology available to them, they had forgotten that you can use a rope to pull someone up.

Consider the lieutenant's first idea. Possibly GPT-2 might have been able to notice that statistically, firefighters typically use rescue harnesses in situations like this. But it doesn't do any mental simulation to see what the predicted outcome of using that harness would be. If there had been enough previous situations where a harness was unusable, and enough cues to indicate to GPT-2 that this was one of those situations, then it could accurately predict that the rescuers would do something different. But if this is a novel situation (as most situations are), then it needs to actually do causal reasoning and notice that the woman would slide off the support. (This is similar to your "check for badness" thing, except it happens via mental simulation rather than just association.)

Orthonormal gives us a real-life example of what happens when an AI uses pattern-matching, but does not do causal reasoning, and then tries to play Starcraft:

The overhyped part is that AlphaStar doesn't really do the "strategy" part of real-time strategy. Each race has a few solid builds that it executes at GM level, and the unit control is fantastic, but the replays don't look creative or even especially reactive to opponent strategies.
That's because there's no representation of causal thinking - "if I did X then they could do Y, so I'd better do X' instead". Instead there are many agents evolving together, and if there's an agent evolving to try Y then the agents doing X will be replaced with agents that do X'. But to explore as much as humans do of the game tree of viable strategies, this approach could take an amount of computing resources that not even today's DeepMind could afford.
(This lack of causal reasoning especially shows up in building placement, where the consequences of locating any one building here or there are minor, but the consequences of your overall SimCity are major for how your units and your opponents' units would fare if they attacked you. In one comical case, AlphaStar had surrounded the units it was building with its own factories so that they couldn't get out to reach the rest of the map. Rather than lifting the buildings to let the units out, which is possible for Terran, it destroyed one building and then immediately began rebuilding it before it could move the units out!)

AlphaStar notices that the units are trapped, which it associates with "must destroy the thing that is trapping them". Then it notices that it is missing a factory, and its associations tell it that in this situation it should have one more factory, and it should be located right where the destroyed factory should be, so...

In contrast, a human might have considered destroying the factory, but then noticed that this leads to a situation where there is one factory too little; and then realized that the building can just be lifted out of the way.

Here is Klein's illustration of how a generic mental simulation seems to work; he also has illustrations of the more specific variant of explaining the past (e.g. the firefighter commander constructing a story of how the basement fire had spread) and projecting to the future (e.g. the firefighter lieutenant trying to figure out how to rescue the woman). Here's him explaining a part of the figures:

Consider this example. Some need arises for building a mental simulation; let us say a coworker has suddenly started acting rudely toward you. The simulation has to let you infer what the original situation was that led to the events you are observing. You assemble the action sequence: the set of transitions that make up the simulation. Perhaps you recall an incident that same morning when you were chatting with some other people in your office and said something that made them laugh. Perhaps you also recall that earlier that morning, your coworker had confided some embarrassing secret to you. So you construct a sequence in which your coworker trusts you with a confidence, then regrets it immediately afterward and feels a little awkward around you, then sees you possibly entertaining some other people with the secret, and then feels that it is going to be unbearable to live with you in the same setting. Now you can even remember that after you made the other people laugh, you looked up and saw the coworker giving you a look that made you feel uneasy. This set of states and transitions is the action sequence, the mental simulation that explains the rude behavior.

The next step is to evaluate the action sequence at a surface level. Is it coherent (Do the steps follow from each other)? Yes, it is. Does it apply (Does the sequence account for the rudeness)? Yes, it does. How complete is it (Does it leave out any important factors, such as the excellent performance evaluation you have just received)? Yes, there are some more pieces that might belong to the puzzle. But in general, the mental simulation passes the internal evaluation. It is an acceptable explanation. That does not mean it is correct.

Sometimes the mental simulation will not pass the internal evaluation, and that also helps you make sense of things. [The following example] illustrates this with a story reported in a newspaper. [...]

The IRA Terrorist. A well-respected lawyer has agreed to defend a man accused of committing an act of terrorism: planting a bomb for the IRA. The lawyer, asked why he would take the case, answers that he interviewed the accused man, who was shaking and literally overcome with panic. He was surprised to see the man fall apart like that. He tried to imagine the IRA's recruiting such a person for a dangerous mission and found that he could not. He cannot conjure up a scenario in which the IRA would give a terrorism assignment to a man like this, so his conclusion is that the man is innocent.

This lawyer could not generate an action sequence that passed his internal evaluation-specifically, the requirement that the transition between steps be plausible. His failure to assemble a plausible sequence of steps led him to a different explanation than the prosecutors had formed. That's why you see a long, curving arc in figure 5.4: the failure to assemble the mental simulation was the basis of the conclusion.

There are also times when you use mental simulation to try to increase your understanding of situations like these. You are trying to build up better models. When you run the action sequence in your mind, you may notice parts that still seem vague. Maybe you can figure out how to set up a better action sequence, or maybe there are some more details about the present state that you should gather. Going back to the example of your coworker, your explanation has not included the fact that you received such a good performance evaluation. What was your coworker's performance evaluation? Possibly the coworker felt you had gotten recognition for work that someone else had done. Perhaps you can get a general sense of the situation by talking to your boss. That might give you some more data points for building your explanations.

(If you look at explanations of how GPT's Transformer architecture works [1, 2], you can see that it doesn't do anything like this.)

Comment by kaj_sotala on What's Your Cognitive Algorithm? · 2020-06-21T07:34:44.110Z · score: 4 (2 votes) · LW · GW
I wouldn't be that surprised if GPT-2 was "only" a System 1. But I also wouldn't be that surprised if it naturally developed a System 2 when scaled up, and given more training. I also wouldn't be that surprised if it turned out not to need a System 2.

As steve2152 also noted, System 2 (or more accurately, Type 2) reasoning involves passing the outputs from one Type 1 system to another using working memory resources. Working memory seems to involve several specialized components, including memory storages and executive functions that control and regulate it. If GPT-2 doesn't have those kinds of architectural properties already, it's not going to develop them by just having more training data thrown at it.

Comment by kaj_sotala on [Personal Experiment] One Year without Junk Media: Six-Month Update · 2020-06-20T06:04:14.074Z · score: 4 (2 votes) · LW · GW

Great to have long-term reports! Very interesting results.

Comment by kaj_sotala on The "hard" problem of consciousness is the least interesting problem of consciousness · 2020-06-16T11:29:11.365Z · score: 2 (1 votes) · LW · GW
I guess my beef is that when it's framed as "But why does XYZ system entail qualia?" I infer that even if in the far future I had a SUPER detailed understanding of "tweak this and you get X more units of  experience, if you don't have ABC any experience is impossible, LMN architecture is really helpful, but not necessary" that Chalmers would still be unimpressed and got "But why does any of this lead to qualia?"
Well, I don't actually think he'd say that. If I had that sorta detailed outline I think his mind would be blown and he'd be super excited.
But when I imagine the person who is still going "But why", I'm imagining that they must be thinking of qualia is this isolated, other, and separate thing.

It's a little unclear from this description whether that understanding would actually solve the hard problem or not? Like, if we have a solution for it, then it would obviously be silly for someone to still say "but why"; but if that understanding actually doesn't solve the problem, then it doesn't seem particularly silly to continue asking the question. Whether or not asking it in that situation implies believing that qualia must be divorced from everything else - I couldn't tell without actually seeing an explanation of that understanding.

Comment by kaj_sotala on The "hard" problem of consciousness is the least interesting problem of consciousness · 2020-06-16T11:11:37.908Z · score: 4 (2 votes) · LW · GW

Well, how do those solve the hard problem?

Comment by kaj_sotala on Research snap-shot: question about Global Workspace Theory · 2020-06-16T09:23:19.578Z · score: 4 (2 votes) · LW · GW
In predictive processing, attention is a system that manipulates the confidence intervals on your predictions. Low attention -> wide intervals -> lots of mismatch between prediction and data doesn't register as an error. High attention -> tighter intervals -> slight mismatch leads to error signal. 

Hmm... that sounds a bit different from how I've understood attention in predictive processing to work. AFAIK, it's not that attending to something tightens its confidence interval; it's that things with tight confidence intervals (relevant for the task in question) get more attention.

So bottom-up attention would be computed by the subsystems that were broadcasting content into GNW, and their attention assignments would be implicit in their output. E.g. if you are looking at someone's face and a subsystem judges that the important thing to pay attention to is the fact that they are looking angry, then it would send the message "this person looks angry" to the GNW. And subsystems would have a combination of learned and innate weights for when their messages could grab control of the GNW and dominate it with what they are paying attention to, similar to the salience cues in the basal ganglia that allow some bids to dominate in specific situations.

Top-down attention would be computed by attentional subsystems interfacing with the GNW, to pick out specific signals in the GNW that seemed most useful for the current task, and strengthening those signals.

The GNW seems like it can only broadcast "simple" or "small" things. A single image, a percept, a signal. Something like a hypothesis in the PP paradigm seems like too big and complex a thing to be "sent" on the GNW

Is it? Like, suppose that a subsystem's hypothesis is that you are seeing a person's face; as a result, the image of a face is sent to the GNW. In that case, the single image that is transmitted into the GNW is the hypothesis. That hypothesis being in consciousness may then trigger an error signal due to not matching another subsystem's data, causing an alternative hypothesis to be broadcast into consciousness.

That said, seeing a face usually involves other things than just the sight of the face: thoughts about the person in question, their intentions, etc. My interpretation has been that once one subsystem has established that "this is a face" (and sends into consciousness a signal that highlights the facial features that it has computed to get the most attention), other subsystems then grab onto those features and send additional details and related information into consciousness. The overall hypothesis is formed by many distinct pieces of data submitted by different subsystems - e.g. "(1) I'm seeing a face, (2) which belongs to my friend Mary, (3) who seems to be happy; (4) I recall an earlier time when Mary was happy".

Here's an excerpt from Consciousness and the Brain that seems relevant:

In 1959, the artificial intelligence pioneer John Selfridge introduced another useful metaphor: the “pandemonium.” He envisioned the brain as a hierarchy of specialized “daemons,” each of which proposes a tentative interpretation of the incoming image. Thirty years of neurophysiological research, including the spectacular discovery of visual cells tuned to lines, colors, eyes, faces, and even U.S. presidents and Hollywood stars, have brought strong support to this idea. In Selfridge’s model, the daemons yelled their preferred interpretation at one another, in direct proportion to how well the incoming image favored their own interpretation. Waves of shouting were propagated through a hierarchy of increasingly abstract units, allowing neurons to respond to increasingly abstract features of the image—for instance, three daemons shouting for the presence of eyes, nose, and hair would together conspire to excite a fourth daemon coding for the presence of a face. By listening to the most vocal daemons, a decision system could form an opinion of the incoming image—a conscious percept.

Selfridge’s pandemonium model received one important improvement. Originally, it was organized according to a strict feed-forward hierarchy: the daemons bellowed only at their hierarchical superiors, but a high-ranking daemon never yelled back at a low-ranking one or even at another daemon of the same rank. In reality, however, neural systems do not merely report to their superiors; they also chat among themselves. The cortex is full of loops and bidirectional projections. Even individual neurons dialogue with each other: if neuron α projects to neuron β, then β probably projects back to α. At any level, interconnected neurons support each other, and those at the top of the hierarchy can talk back to their subordinates, so that messages propagate downward at least as much as upward.

Simulation and mathematical modeling of realistic “connectionist” models with many such loops show that they possess a very useful property. When a subset of neurons is excited, the entire group self-organizes into “attractor states”: groups of neurons form reproducible patterns of activity that remain stable for a long duration. As anticipated by Hebb, interconnected neurons tend to form stable cell assemblies.

As a coding scheme, these recurrent networks possess an additional advantage—they often converge to a consensus. In neuronal networks that are endowed with recurrent connections, unlike Selfridge’s daemons, the neurons do not simply yell stubbornly at one another: they progressively come to an intelligent agreement, a unified interpretation of the perceived scene. The neurons that receive the greatest amount of activation mutually support one another and progressively suppress any alternative interpretation. As a result, missing parts of the image can be restored and noisy bits can be removed. After several iterations, the neuronal representation encodes a cleaned-up, interpreted version of the perceived image. It also becomes more stable, resistant to noise, internally coherent, and distinct from other attractor states. Francis Crick and Christof Koch describe this representation as a winning “neural coalition” and suggest that it is the perfect vehicle for a conscious representation.

The term “coalition” points to another essential aspect of the conscious neuronal code: it must be tightly integrated. Each of our conscious moments coheres as one single piece. When contemplating Leonardo da Vinci’s Mona Lisa, we do not perceive a disemboweled Picasso with detached hands, Cheshire cat smile, and floating eyes. We retrieve all these sensory elements and many others (a name, a meaning, a connection to our memories of Leonardo’s genius)—and they are somehow bound together into a coherent whole. Yet each of them is initially processed by a distinct group of neurons, spread centimeters apart on the surface of the ventral visual cortex. How do they get attached to one another?

One solution is the formation of a global assembly, thanks to the hubs provided by the higher sectors of cortex. These hubs, which the neurologist Antonio Damasio calls “convergence zones,” are particularly predominant in the prefrontal cortex but also in other sectors of the anterior temporal lobe, inferior parietal lobe, and a midline region called the precuneus. All send and receive numerous projections to and from a broad variety of distant brain regions, allowing the neurons there to integrate information over space and time. Multiple sensory modules can therefore converge onto a single coherent interpretation (“a seductive Italian woman”). This global interpretation may, in turn, be broadcast back to the areas from which the sensory signals originally arose. The outcome is an integrated whole. Because of neurons with long-distance top-down axons, projecting back from the prefrontal cortex and its associated high-level network of areas onto the lower-level sensory areas, global broadcasting creates the conditions for the emergence of a single state of consciousness, at once differentiated and integrated.

This permanent back-and-forth communication is called “reentry” by the Nobel Prize winner Gerald Edelman. Model neuronal networks suggest that reentry allows for a sophisticated computation of the best possible statistical interpretation of the visual scene. Each group of neurons acts as an expert statistician, and multiple groups collaborate to explain the features of the input. For instance, a “shadow” expert decides that it can account for the dark zone of the image—but only if the light comes from the top left. A “lighting” expert agrees and, using this hypothesis, explains why the top parts of the objects are illuminated. A third expert then decides that, once these two effects are accounted for, the remaining image looks like a face. These exchanges continue until every bit of the image has received a tentative interpretation.
Comment by kaj_sotala on Research snap-shot: question about Global Workspace Theory · 2020-06-16T08:35:34.091Z · score: 4 (2 votes) · LW · GW
If you put three targets all in a row, people are able to detect them just fine. Additionally, if you ask people to remember the entire sequence they can do better than when you ask them to just remember only some of the characters (up till the point where you max out working memory). This makes no sense if the earlier experiments where interacting with a fundamental processing period that anything being attended to requires. 

Huh! Nice find. That's weird, I'm confused now.

Comment by kaj_sotala on Research snap-shot: question about Global Workspace Theory · 2020-06-16T08:21:04.803Z · score: 2 (1 votes) · LW · GW
Easy tasks can route around the global workspace, hard ones or ones that produce error have to go through it. That's the previous idea. Now, this paper has begun to shift my thinking. For a specific set of tasks, it claims to show that training doesn't shift activity away from a bottleneck location, but instead makes the processing at the point of the bottleneck more efficient.

Interesting! This would make a lot of intuitive sense - after the subsystems responsible for some task have been sufficiently trained, they can mostly just carry out the task using their trained-up predictive models, and need a lot less direct sensory data.

This might also explain some aspects of meditation: for example, The Mind Illuminated talks about "increasing the power of consciousness", which it describes as literally increasing the amount of experience-moments per unit of time. I was never quite sure of how exactly to explain that in terms of a global workspace model... but maybe if the system for generating moments of introspective awareness also got more efficient somehow? Hmm...

If subsystems had to route through the GNW to trigger motor actions, then this system or some variation could totally account for the serial conflict resolution function. But if subsystems can directly send motor commands without going through the GNW, how would would subsystems in conflict be "told to stop" while the conflict resolution happens? The GNW is not a commander, it can't order subsystems around. Though it may be central to consciousness, it's not the "you" that commands and thinks.
All this leaves me thinking that I'm either missing a big obvious chunk of research, or that various motor-planning parts of the brain can't send motor commands except via the GNW. Please point me at any relevant research that you know of.

So AFAIK, the command bottleneck is in the basal ganglia, which are linked to the GNW. A lot of the brain works by lateral inhibition, where each of neurons A, B and C may fire, but any of them firing sends inhibitory signals to the others, causing only one of them to be active at a time.

My understanding from the scholarpedia article is that something similar is going on in the basal ganglia - different subsystems send various motor command "bids" to the BG, which then get different weights depending on various background factors (e.g. food-seeking behaviors get extra weight when you are hungry). Apparently there's a similar mechanism of a strong bid for one system inhibiting the bidding signals for all the others. So if multiple subsystems are issuing conflicting bids at the same time, their bids would end up inhibiting each other and none of them would reach the level necessary for carrying out actions.

Scholarpedia links to Prescott et al. 2006 (sci-hub) as offering a more detailed model, including a concrete version that was implemented in a robot. I've only skimmed it, but they note that their model has some biologically plausible behaviors. In some situations where the robot experiences conflicting high-weight bids, it seems to "react to uncertainty by going half-speed": doing a bit of one one response and then a bit of another, and failing to fully commit to any single procdure:

The control architecture of the robot includes five behaviors, or action sub-systems, which it can switch between at any time. These are: searching for cylinders (cylinder-seek or Cs), picking up a cylinder (cylinderpickup, Cp), looking for a wall (wall-seek, Ws), following alongside a wall (wall-follow, Wf), and depositing the cylinder in a corner (cylinder-deposit, Cd). [...]

three of the five action sub-systems—cylinder-seek, wall-seek, and wall-follow— map patterns of input from the peripheral sensor array into movements that orient the robot towards or away from specific types of stimuli (e.g. object contours). [...]

By comparison, the fixed action pattern for cylinder-pickup in the robot model constitutes five sequential elements: (i) slowly approach the cylinder (to ensure correct identification and good alignment for pickup), (ii) back away (to allow room to lower the arm) whilst opening the gripper, (iii) lower the arm to floor level, (iv) close the gripper (hopefully around the cylinder), (v) return the arm to vertical. [...]

A centralized action selection system requires mechanisms that can assimilate relevant perceptual, motivational, and contextual signals to determine, in some form of ‘common currency’ the relative salience or urgency of each competing behavior (McFarland, 1989; Redgrave et al., 1999a). In the embedding architecture of our model, at each time-step, a salience value for each action sub-system is calculated as a weighted sum of relevant perceptual and motivational variables, and may also be dependant on the current activity status of the action sub-system itself. [...]

... a breakdown of clean selection can occur in the disembodied model when two competitors have high salience levels. To examine the behavioral consequences of this pattern of selection, the robot model was tested over five trials of 120s (approximately 800 robot time-steps) in which the salience of every channel was increased, on every time-step, by a constant amount (C0.4) [...]

During a continuous sequence of high salience competitions the robot exhibited patterns of behavioral disintegration characterized by: (i) reduced efficiency and distortion of a selected behavior, and (ii) rapid switching and incomplete foraging behavior.

Fig. 11 shows the effect of increased salience intensity on exploration of the winner/most-salient-loser salience-space over all trials. The graph demonstrates that virtually all (w4000) salience competitions appeared in the region of salience space (compare with Fig. 7) where reduced efficiency and distorted selection can be expected

The initial avoidance sequence followed the expected pattern but the transition to foraging activity did not begin cleanly, instead showing reduced efficiency and intermittent, partial selection of (losing) avoidance behaviors. To the observer the movement of the robot behavior during the transition appeared somewhat slowed and ‘tentative’. During the foraging bout there was an extended period of rapid switching between cylinder-seek and cylinder-pickup with the robot repeatedly approaching the cylinder but failing to grasp it. The pattern initially observed (t=60–85s) was for the robot to approach the cylinder; back up as if to collect it in the gripper; then move forward without lowering the gripper-arm, pushing the cylinder forward slightly. Later (t=85–90s, 110–115s), where both behaviors showed some partial selection, the robot would lower the arm whilst moving forward but fail to grasp the cylinder due to being incorrectly aligned.

In all five trials, the selection behavior of the robot was similarly inefficient and distorted with the robot frequently displaying rapid alternation of foraging acts. This is illustrated in the transition matrix in Table 2, which shows that the behavior of the robot was dominated by the sequence Cs–Cp–Cs–Cp. with no trials leading to a successful foraging sequence. [...]
Whilst the performance of the model in these circumstances is clearly sub-optimal from a purely action selection viewpoint, it shows interesting similarities to the findings of a large number of studies investigating the behavior of animals in conflict situations (Fentress, 1973; Hinde, 1953, 1966; Roeder, 1975). For instance, Hinde (1966) describes a number of possible outcomes that have been observed in ethological studies of strong behavioral conflicts: (i) inhibition of all but one response, (ii) incomplete expression of a behavior (generally the preparatory stages of behavior are performed), (iii) alternation between behaviors (or ‘dithering’), (iv) ambivalent behavior (a mixture of motor responses), (v) compromise behavior (similar to ambivalence, except that the pattern of motor activity is compatible with both behavioral tendencies), (vi) autonomic responses (for instance defecation or urination), (vii) displacement activity (expression of a behavior that seems irrelevant to the current motivational context, e.g. grooming in a ‘fight or flight’ conflict situation) . Of these outcomes, several show clear similarities with the behavior of the robot in the high salience condition. Specifically, the distortion observed in the early stages of the trial could be understood as a form of ambivalent behavior (iv), whilst the later repetitive behavioral switching has elements of both incomplete expression of behavior (ii) and alternation (iii).

More generally, the behavior of the embodied basal ganglia model is consistent a wide range of findings in psychology and ethology demonstrating that behavioral processes are most effective at intermediate levels of activation (Berlyne, 1960; Bindra, 1969; Fentress, 1973; Malmo, 1959), These findings can also be viewed as expressing the Yerkes-Dodson law (Yerkes & Dodson, 1908) that predicts an ‘inverted U’-shaped relationship between arousal and performance. Our model is consistent with this law in that the robot shows little or no behavioral expression when only low salience inputs are present, demonstrates effective action selection for a range of intermediate level salience inputs (and for high salience inputs where there is no high salience competitor), and exhibits disintegrated behavior in circumstance of conflict between multiple high-salience systems. The robot model, therefore, suggests that the basal ganglia form an important element of the neural substrate mediating the effects of arousal on behavioral effectiveness.

Connecting this with the GNW, several of the salience cues used in the model are perceptual signals, e.g. whether or not a wall or a cylinder is currently perceived. We also know that signals which get to the GNW have a massively boosted signal strength over ones that do not. So while the GNW does not "command" any particular subsystem to take action, salience cues that get into the GNW can get a significant boost, helping them win the action selection process.

Compare e.g. the Stanford Marshmallow Experiment, where the children used a range of behaviors to distract themselves from the sight/thought of the marshmallow - or any situation where you yourself keep getting distracted by signals making their way to consciousness:

The three separate experiments demonstrate a number of significant findings. Effective delay of gratification depends heavily on the cognitive avoidance or suppression of the reward objects while waiting for them to be delivered. Additionally, when the children thought about the absent rewards, it was just as difficult to delay gratification as when the reward items were directly in front of them. Conversely, when the children in the experiment waited for the reward and it was not visibly present, they were able to wait longer and attain the preferred reward. The Stanford marshmallow experiment is important because it demonstrated that effective delay is not achieved by merely thinking about something other than what we want, but rather, it depends on suppressive and avoidance mechanisms that reduce frustration.
The frustration of waiting for a desired reward is demonstrated nicely by the authors when describing the behavior of the children. “They made up quiet songs…hid their head in their arms, pounded the floor with their feet, fiddled playfully and teasingly with the signal bell, verbalized the contingency…prayed to the ceiling, and so on. In one dramatically effective self-distraction technique, after obviously experiencing much agitation, a little girl rested her head, sat limply, relaxed herself, and proceeded to fall sound asleep.”
Comment by kaj_sotala on We've built Connected Papers - a visual tool for researchers to find and explore academic papers · 2020-06-15T11:03:48.720Z · score: 2 (1 votes) · LW · GW

Seems hard since there's much less citing, and even posts that reference earlier ones don't necessarily link to them (e.g. people often talk about "slack" without linking to the original post).

Comment by kaj_sotala on Status - is it what we think it is? · 2020-06-14T12:41:48.609Z · score: 2 (1 votes) · LW · GW

Another relevant excerpt, from siderea's analysis of Watership Down:

One of the things about being the King for a people, is that you get blamed. Even for things that aren't your fault. Even for things beyond your control. Even for crappy-ass reasons like, "I'm scared and pissed off and you, you're in charge here, so I'll vent my feelings against you."
This is the challenging part of caring. If you demonstrate a concern for the wellbeing of the people in your people, they will start seeing their wellbeing as your concern. Start taking responsibility for how things go in a group, and people will start seeing you as responsible for how things go in a group.
This, right here, is what causes many people to back away from Kingship. Which is their right, of course. It's totally legitimate to look at that deal and say, "Oh, hell no."
Our society tells us that being King is awesome and everyone – well, everyone normal – wants to be one. "Every body wants to rule the world." No, actually, they don't. My experience tells me that most people are very reluctant to step into the job of King, and this consequence of the role is a primary reason why. People who, even knowing this consequence, are still willing to have authority rest on their shoulders are not at all that common.
Comment by kaj_sotala on The "hard" problem of consciousness is the least interesting problem of consciousness · 2020-06-08T20:00:07.151Z · score: 5 (3 votes) · LW · GW
So when I hear a claim that "subjective experience" and "qualia" are divorced from any and all behavior or functionality in the mind

I don't think that Chalmers would claim that they are. He's only saying that there doesn't seem to be any obvious logical reason for why someone would need to have subjective experience, and that it needs to be explained why we seem to have subjective experience anyway.

When you say:

The more I think about qualia, the more I feel like the only meaning I can find in any of my subjective experiences is in how they relate to everything else in my head.

Then one could answer "but why couldn't a computer system just have this enum-like system that had all the properties which match your subjective experience, without having that subjective experience?"

Note that this is not claiming that your subjective experiences wouldn't be related to the behavior and functionality of your mind. They obviously are! But that doesn't explain why they are.

Comment by kaj_sotala on From self to craving (three characteristics series) · 2020-06-05T07:42:54.364Z · score: 2 (1 votes) · LW · GW
If a model were trying to prevent itself from being falsified, that would predict that we look away from things that we're not sure about rather than towards them.

That sounds like the dark room problem. :) That kind of thing does seem to sometimes happen, as people have varying levels of need for closure. But there seem to be several competing forces going on, one of them being a bias towards proving the hypothesis true by sampling positive evidence, rather than just avoiding evidence.

Model A: I will eat a cookie and this will lead to an immediate reward associated with the sweet taste
Model B: I won't eat the cookie, instead I'll meditate on gratitude and this will make me very happy
Now in my perspective, this is great evidence that valence and reward are two different things. If becoming happy is the same as reward, why haven't I meditated in the last 5 years even though I know it makes me happy? And why do I want to eat that cookie even though I totally understand that it won't make me smile even while I'm eating it, or make me less hungry, or anything?

This is actually a nice example, because I claim that if you learn and apply the right kinds of meditative techniques and see the craving in more detail, then your mind may notice that eating the cookie actually won't bring you very much lasting satisfaction (even if it does bring a brief momentary reward)... and then it might gradually shift over to preferring meditation instead. (At least in the right circumstances; motivation is affected by a lot of different factors.)

Which cravings get favored in which circumstances looks like a complex question, that I don't have a full model of... but we know from human motivation in general that there's a bias towards actions that bring immediate rewards. To some extent it might be a question of the short-term rewards simply getting reinforced more. Eating a cookie takes less time than meditating for an hour, so if you are more likely to eat more cookies than you finish meditation sessions, each eaten cookie slightly notching up the estimated confidence in the hypothesis and biasing your future decisions even more in favor of the cookie.

The thing is, Model A is totally correct; eating the cookie would lead to an immediate reward! It doesn't need to distort anything, as far as it goes.

So the prediction that craving makes isn't actually "eating the cookie will bring reward"; I'm not sure of what the very exact prediction is, but it's closer to something like "eating the cookie will lead to less dissatisfaction". And something like the following may happen:

You're trying to meditate, and happen to think of the cookie on your desk. You get a craving to stop meditating and go eat the cookie. You try to resist the craving, but each moment that you resist it feels unpleasant. Your mind keeps telling you that if you just gave in to the temptation, then the discomfort from resisting it would stop. Finally, you might give in, stopping your meditation session short and going to eat the cookie.

What happened here was that the craving told you that in order to feel more satisfied, you need to give in to the craving. When you did go eat the cookie, this prediction was proven true. But there was a self-fulfilling prophecy there: the craving told you that the only way to eliminate the discomfort was by giving in to the craving, when just dropping the craving would also have eliminated the discomfort. Maybe the craving didn't exactly distort the sense data, but it certainly sampled a very selected part of it.

The reason why I like to think of cravings as hypotheses, is that if you develop sufficient introspective awareness for the mind to see in real time that the craving is actively generating discomfort rather than helping avoid it, (that particular) craving will be eliminated. The alternative hypothesis that replaces it is then something like "I'm fine even if I go without a cookie for a while".

Comment by kaj_sotala on On the construction of the self · 2020-06-03T14:28:51.894Z · score: 5 (3 votes) · LW · GW

Yeah. I find that many experiences that I've had in meditation-induced states are basically impossible to recall precisely: I will only remember a surface impression or verbal summary of what the state felt like. Then when I get back to that state, I have an experience of "oh yeah, this is what those words that I wrote down earlier really meant".

Comment by kaj_sotala on Building brain-inspired AGI is infinitely easier than understanding the brain · 2020-06-03T10:02:48.476Z · score: 4 (2 votes) · LW · GW
If some circuit in the brain is doing something useful, then it's humanly feasible to understand what that thing is and why it's useful, and to write our own CPU code that does the same useful thing.
In other words, the brain's implementation of that thing can be super-complicated, but the input-output relation cannot be that complicated—at least, the useful part of the input-output relation cannot be that complicated.

Robin Hanson makes a similar argument in "Signal Processors Decouple":

The bottom line is that to emulate a biological signal processor, one need only identify its key internal signal dimensions and their internal mappings – how input signals are mapped to output signals for each part of the system. These key dimensions are typically a tiny fraction of its physical degrees of freedom. Reproducing such dimensions and mappings with sufficient accuracy will reproduce the function of the system.
This is proven daily by the 200,000 people with artificial ears, and will be proven soon when artificial eyes are fielded. Artificial ears and eyes do not require a detailed weather-forecasting-like simulation of the vast complex physical systems that are our ears and eyes. Yes, such artificial organs do not exactly reproduce the input-output relations of their biological counterparts. I expect someone with one artificial ear and one real ear could tell the difference. But the reproduction is close enough to allow the artificial versions to perform most of the same practical functions.
We are confident that the number of relevant signal dimensions in a human brain is vastly smaller than its physical degrees of freedom. But we do not know just how many are those dimensions. The more dimensions, the harder it will be to emulate them. But the fact that human brains continue to function with nearly the same effectiveness when they are whacked on the side of the head, or when flooded with various odd chemicals, shows they have been designed to decouple from most other physical brain dimensions.
The brain still functions reasonably well even flooded with chemicals specifically designed to interfere with neurotransmitters, the key chemicals by which neurons send signals to each other! Yes people on “drugs” don’t function exactly the same, but with moderate drug levels people can still perform most of the functions required for most jobs.
Comment by kaj_sotala on Conceptual engineering: the revolution in philosophy you've never heard of · 2020-06-03T05:03:25.187Z · score: 17 (7 votes) · LW · GW

As part of that larger project, I want to introduce a frame that, to my knowledge, hasn't yet been discussed to any meaningful extent on this board: conceptual engineering, and its role as a solution to the problems of "counterexample philosophy" and "conceptual analysis"—the mistaken if implicit belief that concepts have "necessary and sufficient" conditions—in other words, Platonic essences.

After reading the essay, I'm still confused by what conceptual engineering actually is. Is it a claim about how humans use language in general, a philosophical technique like conceptual analysis is, or both?

(You seemed to attack Chalmers for trying to offer a definition for conceptual engineering, but a brief definition for the concept was exactly what I found myself hoping for. I think you are saying that you don't want to offer one because terms don't have necessary and sufficient definitions so offering a definition goes against the whole spirit of the approach... but you also note that we learn words by seeing a specific example and expanding from that, so I wouldn't think that it would be contrary to the spirit of the approach to offer a brief definition and then expand from that once the readers have something to hang their minds on.)

Comment by kaj_sotala on Spoiler-Free Review: Monster Train · 2020-06-02T14:27:48.437Z · score: 2 (1 votes) · LW · GW

25 hours played, this game has been growing on me as well. Though I too would like it if there was more room for creative improvisation, as in Slay the Spire, rather than strict optimization. A lot of the units have just outright fun concepts, and it would be nice if mixing and matching them more freely would make gameplay sense. Often I just do so anyway, even if it means that I'm less likely to win. (This might explain why I still mostly lose even on Covenant Rank 1.)

An interesting thing that I've noticed, and people seem to agree with, is that the game feels shorter than Slay the Spire despite one run taking about the same time in real-world minutes. (My successful StS runs are about 1-2 hours; Monster Train, about an hour and a half.) When I finished my first run, I assumed that it was just the first world, since it felt roughly like the end of the first act in StS. That's an interesting psychological observation by itself. Seems to be a result of the upgrade-battle-upgrade loop having been reduced to a much smaller number of battles and upgrade points, but with those points being correspondingly larger.

Comment by kaj_sotala on A mechanistic model of meditation · 2020-06-02T09:44:58.979Z · score: 2 (1 votes) · LW · GW

Thank you! I'm happy to hear that.

Mostly because of the possibility of suppressing undesired content, and because around Stage 6 was where I noticed myself picking up the ability to just completely exclude unwanted parts of my mind from consciousness. A friend who did TMI all the way to stage 10 also said that roughly this stage was the region where it might get tricky.

To be clear, it doesn't seem to me that TMI past this stage would automatically cause harm: it's just something to be aware of, in case you notice yourself suppressing subminds rather than taking their concerns into account. A TMI teacher who I spoke with said that the most typical outcome of trying to use the system to suppress some of your desires is that you just stop making progress or revert to an earlier stage. The mind does have its defenses against this kind of a thing.

Not sure what you're asking about jhana practices, but I have very little personal experience with them, and I haven't heard them described as having the same problems. (Especially since people typically don't do strong jhanas off the couch, and suppressing subminds in your daily life is the part that seems most risky to me. If it's just that you concentrate really deep during formal meditation, that's probably not an issue.)

Comment by kaj_sotala on On the construction of the self · 2020-06-02T09:25:32.328Z · score: 2 (1 votes) · LW · GW

Thanks! You could say that, yes.

A lot of this already draws on neuroscience, particularly neuronal workspace theory, but I agree that there's still a lot more that could be brought in. Appreciate the book suggestion, it looks interesting.

Comment by kaj_sotala on On the construction of the self · 2020-06-02T09:22:32.143Z · score: 2 (1 votes) · LW · GW
This would also explain Julian Jaynes's seemingly-crazy theory that ancient people experienced their gods as auditory hallucinations. If society tells you that some of the thoughts that you hear in your head come from gods, then maybe your narrative of the self just comes to assign those thoughts as coming from gods.

Speak of the devil.

Comment by kaj_sotala on Paul Crowley's Shortform · 2020-06-01T15:39:17.862Z · score: 2 (1 votes) · LW · GW

Huh. Wow.

Comment by kaj_sotala on On the construction of the self · 2020-06-01T10:05:27.108Z · score: 5 (3 votes) · LW · GW
I'm really enjoying all these posts, thanks a lot!

Thank you for saying that. :)

Wouldn't it be simpler to say that righteous indignation is a rewarding feeling (in the moment) and we're motivated to think thoughts that bring about that feeling?

Well, in my model there are two layers:

1) first, the anger is produced by a subsystem which is optimizing for some particular goal

2) if that anger looks like it is achieving the intended goal, then positive valence is produced as a result; that is experienced as a rewarding feeling that e.g. craving may grab hold of and seek to maintain

That said, the exact reason for why anger is produced isn't really important for the example and might just be unnecessarily distracting, so I'll remove it.

Agreed, and this is one of the reasons that I think normal intuitions about how agents behave don't necessarily carry over to self-modifying agents whose subagents can launch direct attacks against each other, see here.

Agreed.

Yeah, just like every other subsystem right?

Yep! Well, some subsystems seem to do actual forward planning as well, but of course that planning is based on cached models.

Comment by kaj_sotala on On the construction of the self · 2020-06-01T09:58:19.857Z · score: 4 (2 votes) · LW · GW

This makes sense to me; would also be compatible with the model of craving as a specifically socially evolved motivational layer.

The thing we call "self" or "consciousness" is not the agent, is not even a subroutine inside the agent, it is the explanation. This is because any time someone describes eir internal experiences, ey are actually describing this "innate narrative": after all, this is exactly its original function.

Yes. Rephrasing it slightly, anything that we observe in the global workspace is an output from some subsystem; it is not the subsystem itself. Likewise, the sense of a self is a narrative produced by some subsystem. This narrative is then treated as an ontologically basic entity, the agent which actually does things, because the subsystems that do the self-modeling can only see the things that appear in consciousness. Whatever level is the lowest that you can observe, is the one whose behavior you need to take as an axiom; and if conscious experience is the lowest level that you can observe, then you take the narrative as something whose independent existence has to be assumed.

(Now I wonder about that self-representational blink associated with the experience of the self. Could it be that the same system which produces the narrative of the self also takes that narrative as input - and that the blink obscures it from noticing that it is generating the very same story which it is basing its inferences on? "I see a self taking actions, so therefore the best explanation must be that there is a self which is taking actions?")

This would also explain Julian Jaynes's seemingly-crazy theory that ancient people experienced their gods as auditory hallucinations. If society tells you that some of the thoughts that you hear in your head come from gods, then maybe your narrative of the self just comes to assign those thoughts as coming from gods.

Comment by kaj_sotala on From self to craving (three characteristics series) · 2020-06-01T08:42:19.889Z · score: 2 (1 votes) · LW · GW
Notably, hypothesizing the other parts doesn't seem to make sense from an evolutionary POV, as it is reasonable to assume that the ability to have "urges" must logically precede the ability to make predictions about the urges, vs. the urges themselves encoding predictions about the outside world. If we have evolved an urge to do something, it is because evolution already "thinks" it's probably a good idea to do the thing, and/or a bad idea not to, so another mechanism that merely recapitulates this logic would be kind of redundant.

A hypothesis that I've been considering, is whether the shift to become more social might have caused a second layer of motivation to evolve. Less social animals animals can act purely based on physical considerations like the need to eat or avoid a predator, but for humans every action has potential social implications, so needs to also be evaluated in that light. There are some interesting anecdotes like Helen Keller's account suggesting that she only developed a self after learning language. The description of her old state of being sounds like there was just the urge, which was then immediately acted upon; and that this mode of operation then became irreversibly altered:

Before my teacher came to me, I did not know that I am. [...] I cannot hope to describe adequately that unconscious, yet conscious time of nothingness. I did not know that I knew aught, or that I lived or acted or desired. I had neither will nor intellect. I was carried along to objects and acts by a certain blind natural impetus. I had a mind which caused me to feel anger, satisfaction, desire. These two facts led those about me to suppose that I willed and thought. [...] I never viewed anything beforehand or chose it. [...] My inner life, then, was a blank without past, present, or future, without hope or anticipation, without wonder or joy or faith. [...]

I remember, also through touch, that I had a power of association. I felt tactual jars like the stamp of a foot, the opening of a window or its closing, the slam of a door. After repeatedly smelling rain and feeling the discomfort of wetness, I acted like those about me: I ran to shut the window. But that was not thought in any sense. It was the same kind of association that makes animals take shelter from the rain. From the same instinct of aping others, I folded the clothes that came from the laundry, and put mine away, fed the turkeys, sewed bead-eyes on my doll's face, and did many other things of which I have the tactual remembrance. When I wanted anything I liked,—ice-cream, for instance, of which I was very fond,—I had a delicious taste on my tongue (which, by the way, I never have now), and in my hand I felt the turning of the freezer. I made the sign, and my mother knew I wanted ice-cream. I "thought" and desired in my fingers. [...]

I thought only of objects, and only objects I wanted. It was the turning of the freezer on a larger scale. When I learned the meaning of "I" and "me" and found that I was something, I began to think. Then consciousness first existed for me. Thus it was not the sense of touch that brought me knowledge. It was the awakening of my soul that first rendered my senses their value, their cognizance of objects, names, qualities, and properties. Thought made me conscious of love, joy, and all the emotions. I was eager to know, then to understand, afterward to reflect on what I knew and understood, and the blind impetus, which had before driven me hither and thither at the dictates of my sensations, vanished forever.

Would also make sense in light of the observation that the sense of self may disappear when doing purely physical activities (you fall back to the original set of systems which doesn't need to think about the self), the PRISM model of consciousness as a conflict-solver, the way that physical and social reasoning seem to be pretty distinct, and a kind of a semi-modular approach (you have the old primarily physical system, and then the new one that can integrate social considerations on top of the old system's suggestions just added on top). If you squint, the stuff about simulacra also feels kinda relevant, as an entirely new set of implications that diverge from physical reality and need to be thought about on their own terms.

I wouldn't be very surprised if this hypothesis turned out to be false, but at least there's suggestive evidence.

Comment by kaj_sotala on From self to craving (three characteristics series) · 2020-06-01T08:26:04.166Z · score: 3 (2 votes) · LW · GW

Cool, that sounds like a mild no-self state alright. :) Though any strong valence is likely to trigger a self schema and pull you out of it, but it's a question of practice.

Your description kinda reminds me of the approach in Loch Kelly's The Way of Effortless Mindfulness; it has various brief practices that may induce states like the one that you describe. E.g. in this one, you imagine the kind of a relaxing state in which there is no problem to solve and the sense of self just falls away. (Directly imagining a no-self state is hard, because checking whether you are in a no-self state yet activates the self-schema. But if you instead imagine an external state which is likely to put you in a no-self state, you don't get that kind of self-reference, no pun intended.)

First, read this mindful glimpse below. Next, choose a memory of a time you felt a sense of freedom, connection, and well-being. Then do this mindful glimpse using your memory as a door to discover the effortless mindfulness that is already here now.
1. Close your eyes. Picture a time when you felt well-being while doing something active like hiking in nature. In your mind, see and feel every detail of that day. Hear the sounds, smell the smells, and feel the air on your skin; notice the enjoyment of being with your companions or by yourself; recall the feeling of walking those last few yards toward your destination.
2. Visualize and feel yourself as you have reached your goal and are looking out over the wide-open vista. Feel that openness, connection to nature, sense of peace and well-being. Having reached your goal, feel what it’s like when there’s no more striving and nothing to do. See that wide-open sky with no agenda to think about, and then simply stop. Feel this deep sense of relief and peace.
3. Now, begin to let go of the visualization, the past, and all associated memories slowly and completely. Remain connected to the joy of being that is here within you.
4. As you open your eyes, feel how the well-being that was experienced then is also here now. It does not require you to go to any particular place in the past or the future once it’s discovered within and all around.

Recently I've also gotten interested in the Alexander Technique, which seems to have a pretty straightforward series of steps for expanding your awareness and then getting your mind to just automatically do things in a way which feels like non-doing. It also seems to induce the kinds of states that you describe, of just watching oneself work, which I had previously only gotten from meditation.

Can you pick up a ball without trying to pick up the ball? It sounds contradictory, but it turns out that there is a specific behaviour we do when we are “trying”, and this behaviour is unnecessary to pick up the ball.
How is this possible? Well, consider when you’ve picked up something to fiddle with without realising. You didn’t consciously intend for it to end up in your hand, but there it is. There was an effortlessness to it. [...]
But this kind of non-‘deliberate’ effortless action needn’t be automatic and unchosen, like a nervous fiddling habit; nor need it require redirected attention / collapsed awareness, like not noticing you picked up the object. You can be fully aware of what you’re doing, and ‘watch’ yourself doing it, while choosing to do it, and yet still have there be this effortless “it just happened” quality. [...]
Suppose you do actually want to pick up that ball over there. But you don’t want to ‘do’ picking-up-the-ball. The solution is to set an intention.
[1] Have the intention to pick up the ball. [2] Expand your awareness to include what’s all around you, the room, the route to the ball, and your body inside the room. [3] Notice any reactions of trying to do picking-up-the-ball (like “I am going to march over there and pick up that ball”, or “I am going to get ready to stand up so I can go pick up that ball”, or “I am going to approach the ball to pick it up”) — and decline those reactions. [4] Wait. Patiently hold the intention to pick up the ball. Don’t stop yourself from moving — stopping yourself is another kind of ‘doing’ — yet don’t try to deliberately/consciously move. [5] Let movement happen.
Comment by kaj_sotala on From self to craving (three characteristics series) · 2020-05-29T13:45:11.623Z · score: 2 (1 votes) · LW · GW

Cool. :)

There does seem to be some sort of meta-process that you can use to decouple from craving regarding these things, though in my experience it seems to require continuous attention, like an actively inhibitory process. In contrast, the model description you gave made it sound like craving was an active process that one could simply refrain from, and I don't think that's predictively accurate.

An analogy that I might use is that learning to let go of craving, is kind of the opposite of the thing where you practice an effortful thing until it becomes automatic. Craving usually triggers automatically and outside your conscious control, but you can come to gradually increase your odds of being able to notice it, catch it, and do something about it.

"An actively inhibitory process" sounds accurate for some of the mental motions involved. Though merely just bringing more conscious attention to the process also seems to affect it, and in some cases interrupt it, even if you don't actively inhibit it.

If this recalibrates the payoff system, it would make sense within my own model, and resolve the part where I don't see how what you describe could be a truly conscious process, in the way that you made it sound.

Not sure how I made it sound :-) but a good description might be "semi-conscious", in the same sense that something like Focusing can be: you do it, something conscious comes up, and then a change might happen. Sometimes enough becomes consciously accessible that you can clearly see what it was about, sometimes you just get a weird sensation and know that something has shifted, without knowing exactly what.

Okay, I'm off to experiment now. This is exciting!

Any results yet? :)

Comment by kaj_sotala on Paul Crowley's Shortform · 2020-05-29T05:20:44.437Z · score: 10 (2 votes) · LW · GW

Can you give some examples of "LW-style thinking" that they now associate with Cummings?

Comment by kaj_sotala on Why aren’t we testing general intelligence distribution? · 2020-05-26T18:52:26.965Z · score: 14 (6 votes) · LW · GW

It has been some years since I looked at the literature, but if I recall correctly, the problem is that g is defined on a population level rather than on the individual level. You can't directly measure someone's raw g because the raw g is meaningless by itself.

Suppose that you have an intelligence test composed of ten subtests, each of which may earn a person up to 10 points, for a total of 100 points. You give that test to a number of people, and then notice that on average, a person doing well on one of the subtests means that they are likely to do better on the other subtests as well. You hypothesize that this is explained by a "general intelligence factor". You find that if you assign each of your test-takers an "intelligence score", then the average subtest score of all the test-takers who share that level of intelligence is some subtest-dependent constant times their "intelligence score".

Let's say that I was one of the people taking your test. Your equations say that I have a g of 2. Subtest A has a factor loading of 1, so I should get 1 * 2 = 2 points on it. Subtest B has a factor loading of 4.5, so I should get 4.5 * 2 = 9 points on it. It turns out that I actually got 9 points on subtest A and 2 points on subtest B, exactly the opposite pattern than the one you predicted! Does this mean that you have made an error? No, because the factor loadings were only defined as indicating the score averaged over all the test-takers who shared my estimated g, rather than predicting anything definite about any particular person.

This means that I can get a very different score profile from my estimated g would predict, for as long as enough others with my estimated g are sufficiently close to the estimate. So my estimated g is not very informative on an individual level. Compare this to measuring someone's height, where if they are 170 cm on one test, they are going to be 170 cm regardless of how you measure it.

Now suppose that you are unhappy with the subtests that you have chosen, so you throw out some and replace them with new ones. It turns out that the new ones are substantially harder: on the old test, people got an average g of 4, but now they only get an average of 2. How do you compare people's results between the old and the new test? Especially since some people are going to be outliers and perform better on the new test - maybe I get lucky and increase my g to a 3. It's as if we were measuring people's heights in both meters and feet, but rather than one straightforwardly converting to another, two people with the same height in meters might have a different height in feet.

Worse, there's also no reason why the intelligence test would have to have 10 subtests that award 10 points each. Maybe I devise my own intelligence test: it has 6 subtests that give 0-6 points each, 3 subtests that give 2-8 points each, and 4 subtests that give 0-20 points each. The resulting raw score distributions and factor loadings are going to be completely different. How do you compare people's results on your old test, your new test, and my test?

Well, one way to compare them would be to just say that you are not even trying to measure raw g (which is not well-defined for individuals anyway), you are measuring IQ. It seems theoretically reasonable that whatever-it-is-that-intelligence-tests-measure would be normally distributed, because many biological and psychometric quantities are, so you just define IQ as following a normal distribution and fit all the scores to that. Now we can at least say that "Kaj got an IQ of 115 on his own test and an IQ of 85 on Bob's test", letting us know that I'm one standard deviation above the median on my test and one standard deviation below it on your test. That gives us at least some way of telling what the raw scores mean.

Suppose that you did stick to just one fixed test, and measured how the raw scores change over time. This is something that is done - it's how the Flynn effect was detected, as there were increasing raw scores. But there are also problems with that, as seen from all the debates over what the Flynn effect changes actually mean.

Let's say that an intelligence test contains a subtest measuring the size of your vocabulary. The theoretical reason for why vocabulary size is thought to correlate with intelligence is that people learn the meaning of new words by hearing them used in a context. With a higher intelligence, you need to hear a word used fewer times to figure out its meaning, so smarter people will on average have larger vocabularies. Now, words are culturally dependent and people will be exposed to different words just by random chance... but if all of the people who the test was normed on are from the same cultural group, then on average, getting a higher score on the vocabulary subtest is still going to correlate with intelligence.

But suppose that you use exactly the same test for 20 years. The meanings of words change: a word that was common two decades ago might be practically nonexistent today ("phone booth"). Or another might become more common. Or a subtest might measure some kind of abstract reasoning, and then people might start playing puzzle games that feature more abstract logic reasoning. Assuming that people's scores on an IQ test can be decomposed into something like (talent + practice), changes in culture can invalidate intertemporal comparisons by changing the amount of practice people get.

So if someone who was 20 took your test in 2000 and got a raw score of 58, and someone who is 20 takes your test in 2020 and also gets a raw score of 58, this might not indicate that they have the same intelligence either... even if they get exactly the same scores on all the subtests. Periodically re-norming the raw scores helps make them more comparable in this case as well; that way we can at least know that what their ranking relative to other 20-year-olds in the same year was.

Comment by kaj_sotala on Building up to an Internal Family Systems model · 2020-05-26T11:55:58.941Z · score: 2 (1 votes) · LW · GW

Thanks! I got some value out of this training guide, though it's primarily aimed at people who already have some therapy training.

Comment by kaj_sotala on From self to craving (three characteristics series) · 2020-05-26T11:52:50.295Z · score: 3 (2 votes) · LW · GW

I'm not sure if I managed to follow all of this, but at least the first paragraph seems spot-on to me. :)

Comment by kaj_sotala on From self to craving (three characteristics series) · 2020-05-26T11:48:56.764Z · score: 17 (6 votes) · LW · GW

Well, whether or not a model is needlessly complex depends on what it needs to explain. :-)

Back when I started thinking about the nature of suffering, I also had a relatively simple model, basically boiling down to "suffering is about wanting conflicting things". (Upon re-reading that post from nine years back, I see that I credit you for a part of the model that I outlined there. We've been at this for a while. :-)) I still had it until relatively recently. But I found that there were things which it didn't really explain or predict. For example:

  • You can decouple valence and aversion, so that painful sensations appear just as painful as before, but do not trigger aversion.
  • Changes to the sense of self cause changes even to the aversiveness of things that don't seem to be related to a self-model (e.g. physical pain).
  • You can learn to concentrate better by training your mind to notice that it keeps predicting that indulging in a distraction is going to eliminate the discomfort from the distracting urges, but that it could just as well just drop the distraction entirely.
  • There are mental moves that you can make to investigate craving, in such a way which causes the mind to notice that maintaining the craving is actually preventing it from feeling good, and then dropping it.
  • If you can get your mind into states in which there is little or no craving, then those states will feel intrinsically good without regard to their valence.
  • Upon investigation, you can notice that many states that you had thought were purely pleasant actually contain a degree of subtle discomfort; releasing the craving in those states then gets you into states that are more pleasant overall.
  • If you train your mind to have enough sensory precision, you can eventually come to directly observe how the mind carries out the kinds of steps that I described under "Let’s say that there is this kind of a process": an experience being painted with valence, that valence triggering craving, a new self being fabricated by that craving, and so on.

From your responses, it's not clear to me how much credibility you lend to these kinds of claims. If you feel that meditation doesn't actually provide any real insight into how minds work and that I'm just deluded, then I think that that's certainly a reasonable position to hold. I don't think that that position is true, mind you, but it seems reasonable that you might be skeptical. After all, most of the research on the topic is low quality, there's plenty of room for placebo and motivated reasoning effects, introspection is famously unreliable, et cetera.

But ISTM that if you are willing to at least grant that me and others who are saying these kinds of things are not outright lying about our subjective experience... then you need to at least explain how come it seems to us like the urge and the aversion from resisting the urge can become decoupled, or why it seems to us like reductions in the sense of self systematically lead to reductions in the aversiveness of negative valence.

I agree that if I were just developing a model of human motivation and suffering from first principles and from what seems to make evolutionary sense, I wouldn't arrive at this kind of an explanation. "An urge directly combines an itch and the desire to scratch it" would certainly be a much more parsimonious model... but it would then predict that you can't have an urge without a corresponding need to engage in it, and that prediction is contradicted both by my experience and the experience of many others who engage in these kinds of practices.

Comment by kaj_sotala on From self to craving (three characteristics series) · 2020-05-26T11:00:21.103Z · score: 2 (1 votes) · LW · GW

Interesting. For some reason I actually hadn't thought about translating these; and I find myself rejecting each proposed translation with "but the connotations of that one aren't exactly right". Maybe I would just stick with the Pali terms. :)

(Just happened to notice this article which translates craving as "halu", FWIW)

Comment by kaj_sotala on From self to craving (three characteristics series) · 2020-05-25T11:23:29.235Z · score: 2 (1 votes) · LW · GW

Hmm... I think that there's something else going on than just an unhealthily strong motivation, given that craving looks like a hypothesis that can often be disproven - see my reply to pjeby in the other comment.

Comment by kaj_sotala on From self to craving (three characteristics series) · 2020-05-25T11:16:34.929Z · score: 14 (7 votes) · LW · GW

Nice points. To start, there are a few subtleties involved.

One issue, which I thought I had discussed but which I apparently ended up deleting in an editing phase, is that while I have been referring to the Buddhist concept of dukkha as "suffering", there are some issues with that particular translation. I have also been using the term "unsatisfactoriness", which is better in some respects.

The issue is that when we say "suffering", it tends to refer to a relatively strong experience: if you felt a tiny bit of discomfort from your left sock being slightly itchy, many people would say that this does not count as suffering, it's just a bit of discomfort. But dukkha also includes your reaction to that kind of very slight discomfort.

Furthermore, you can even have dukkha that you are not conscious of. Often we think that suffering is a subjective experience, so something that you are conscious of by definition. Can you suffer of something without being conscious of the fact that you are suffering? I can avoid this kind of an issue by saying that dukkha is not exactly the same thing as our common-sense definition of suffering, and unlike the common-sense definition, it doesn't always need to be conscious. Rather, dukkha is something like a training signal that is used by the brain to optimize its functioning and to learn to avoid states with a lot of dukkha: like any other signal in the brain, it has the strongest effect when the signal becomes strong enough to make it to conscious awareness, but it has an effect even if just unconscious.

One example of unconscious dukkha might be this. Sometimes there is a kind of a background discomfort or pain that you have gotten used to, and you think that you are just fine. But once something happens to make that background discomfort go away, you realize how much better you suddenly feel, and that you were actually not okay before.

My model is something like: craving comes in degrees. A lot of factors go into determining how strong it is. Whenever there is craving, there is also dukkha, but if the craving is very subtle, then the dukkha may also be very subtle. There's a spectrum of how easy it is to notice, going roughly something like:

  • Only noticeable in extremely deep states of meditative absorption; has barely any effect on decision-making
  • Hovering near the threshold of conscious awareness, becoming noticeable if it disappears or when there's nothing else going on that could distract you
  • Registers as a slight discomfort, but will be pushed away from consciousness by any distraction
  • Registers as a moderate discomfort that keeps popping up even as other things are going on
  • Experienced as suffering, obvious and makes it hard to focus on anything else
  • Extreme suffering, makes it impossible to think about anything else

So when you say that suffering seems to be most strongly associated with wanting conflicting things, I agree with that... that is, I agree that that tends to produce the strongest levels of craving (by making two strong cravings compete against each other), and thus the level of dukkha that we would ordinarily call "suffering".

At the same time, I also think that there are levels of craving/dukkha that are much subtler, and which may be present even in the case of e.g. imagining a delicious food - they just aren't strong enough to consciously register, or to have any other effect on decision-making; the main influence in those cases is from non-craving-based motivations. (When the craving is that subtle, there's also a conflict, but rather than being a conflict between two cravings, it's a conflict between a craving and how reality is - e.g. "I would like to eat that food" vs. "I don't actually have any of that food right now".)

perhaps what you're saying is that I would have to also think "it would make me happy to eat that, so I should do that in order to be happy."

I think there's something like this going on, yes. I mentioned in my previous post that

a craving for some outcome X tends to implicitly involve at least two assumptions:
1. achieving X is necessary for being happy or avoiding suffering
2. one cannot achieve X except by having a craving for it
Both of these assumptions are false, but subsystems associated with craving have a built-in bias to selectively sample evidence which supports these assumptions, making them frequently feel compelling. Still, it is possible to give the brain evidence which lets it know that these assumptions are wrong: that it is possible to achieve X without having craving for it, and that one can feel good regardless of achieving X.

One way that I've been thinking of this, is that a craving is a form of a hypothesis, in the predictive processing sense where hypotheses drive behavior by seeking to prove themselves true. For example, your visual system may see someone's nose and form the hypothesis that "the thing that I'm seeing is a nose, and a nose is part of a person's face, so I'm seeing someone's face". That contains the prediction "faces have eyes next to the nose, so if I look slightly up and to the right I will see an eye, and if I look left from there I will see another eye"; it will then seek to confirm its prediction by making you look at those spots and verify that they do indeed contain eyes.

This is closely related to two points that you've talked about before; that people form unconscious beliefs about what they need in order to be happy, and that the mind tends to generate filters which pick out features of experience that support the schema underlying the filter - sometimes mangling the input quite severely to make it fit the filter. The "I'm seeing a face" hypothesis is a filter that picks out the features - such as eyes - which support it. In terms of the above, once a craving hypothesis for X is triggered, it seeks to maintain the belief that happiness requires getting X, focusing on evidence which supports that belief. (To be clear, I'm not saying that all filters are created by craving; rather, craving is one subtype of such a filter.)

My model is that the brain has something like a "master template for craving hypotheses". Whenever something triggers positive or negative valence, the brain "tries on" the generic template for craving ("I need to get / avoid this in order to be happy") adapted to this particular source of valence. How strong of a craving is produced, depends on how much evidence can be found to support the hypothesis. If you just imagine a delicious food but aren't particularly hungry, then there isn't much of a reason to believe that you need it for your happiness, so the craving is pretty weak. If you are stressed out and seriously need to get some work done, then "I need to relax while I'm on my walk" has more evidence in its favor, so it produces a stronger craving.

One description for the effects of extended meditative practice is "you suffer less, but you notice it more". Based on the descriptions and my own experience, I think this means roughly the following:

  • By doing meditative practices, you develop better introspective awareness and ability to pay attention to subtle nuances of what's going on in your mind.
  • As your ability to do this improves, you become capable of seeing the craving in your mind more clearly.
  • All craving hypotheses are ultimately false, because they hold that craving is necessary for avoiding dukkha (discomfort), but actually craving is that which generates dukkha in the first place. Each craving hypothesis attributes dukkha to an external source, when it is actually an internally-generated error signal.
  • When your introspective awareness and equanimity sharpen enough, your mind can grab onto a particular craving without getting completely pulled into it. This allows you to see that the craving is trying to avoid discomfort, and that it is also creating discomfort by doing so.
  • Seeing both of these at the same time proves the craving hypothesis false, triggering memory reconsolidation and eliminating the craving.
  • In order to see the craving clearly enough to eliminate it, your introspective awareness had to become sharper and more capable of magnifying subtle signals to the level of conscious awareness. As a result, as you eliminate strong and moderate-strength cravings, the "detection threshold" for when a craving and its associated dukkha is strong enough to become consciously detectable drops. Cravings and discomforts which were previously too subtle to notice, now start appearing in consciousness.
  • The end result is that you have less dukkha (suffering) overall, but become better at noticing those parts of it that you haven't eliminated yet.

There are some similarities between working with craving, and the kind of work with the moral judgment system that you discussed in your post about it. That is, we have learned rules/beliefs which trigger craving in particular situations, just as we have learned rules/beliefs which trigger moral judgment in some situations. As with moral judgment, craving is a system in the brain that cannot be eliminated entirely, and lots of its specific instances need to be eliminated separately - but there are also interventions deeper in the belief network that propagate more widely, eliminating more cravings.

One particular problem with eliminating craving is that even as you eliminate particular instances of it, new craving keeps being generated, as the underlying beliefs about its usefulness are slow to change even as special cases get repeatedly disproven. The claim from Buddhist psychology, which my experience causes me to consider plausible, is that the beliefs which cause cravings to be learned are entangled with beliefs about the self. Changing the beliefs which form the self-model cause changes to craving - as the conception of "I" changes, that changes the kinds of evidence which are taken to support the hypothesis of "I need X to be happy". Drastic enough updates to the self-model can cause a significant reduction in the amount of craving that is generated, to the point that one can unlearn it faster than it is generated.

Though I think that I'm trying to clarify that it is not merely valence or sensation being located in the self, but that another level of indirection is required, as in your "walk to relax" example...

So for craving, indirection can certainly make it stronger, but at its most basic it's held to be a very low-level response to any valence. Physical pain and discomfort is the most obvious example: pain is very immediate and present, but if becomes experienced as less self-related, it too becomes less aversive. In an earlier comment, I described an episode in which my sense of self seemed to become temporarily suspended; the result was that strong negative valence (specifically cold shock from an icy shower) was experienced just as strongly and acutely as before, but it lacked the aversive element - I got out of the shower because I was concerned about the health effects of long-term exposure, but could in principle have remained there for longer if I had wanted. I have had other similar experiences since then, but that one was the most dramatic illustration.

In the case of physical pain, the hypothesis seems to be something like "I have to get this sensation of pain out of my consciousness in order to feel good". If that hypothesis is suspended, one still experiences the sensation of pain, but without the need to get it out of their mind.

(This sometimes feels really weird - you have a painful sensation in your mind, and it feels exactly as painful as always, and you keep expecting yourself to flinch away from it right now... except, you just never do. It just feels really painful and the fact that it feels really painful also does not bother you at all, and you just feel totally confused.)

But the moral judgment system can produce craving/compulsion loops around other people's behavior, without self-reference! You can go around thinking that other people are doing the wrong thing or should be doing something else, and this creates suffering despite there not being any "self" designated in the thought process. (e.g. "Someone is wrong on the internet!" is not a thought that includes a self whose state is to be manipulated, but rather a judgment that the state of the world is wrong and must be fixed.)

So there's a subtlety in that the moral judgment system is separate from the craving system, but it does generate valence that the craving system also reacts to, so their operation gets kinda intermingled. (At least, that's my working model - I haven't seen any Buddhist theory that would explicitly make these distinctions, though honestly that may very well just be because I haven't read enough of it.)

So something like:

  • You witness someone being wrong on the internet
  • The moral judgment system creates an urge to argue with them
  • Your mind notices this urge and forms the prediction that resisting it would feel unpleasant, and even though giving into it isn't necessarily pleasant either, it's at least less unpleasant than trying to resist the urge
  • There's a craving to give in to the urge, consisting of the hypothesis that "I need to give in to this urge and prove the person on the internet wrong, or I will experience greater discomfort than otherwise"
  • The craving causes you to give in to the urge

This is a nice example of how cravings are often self-fulfilling prophecies. Experiencing a craving is unpleasant; when there is negative valence from resisting an urge, craving is generated which tries to resist that negative valence. The negative valence would not create discomfort by itself, but there is discomfort generated by the combination of "craving + negative valence". The craving says that "if I don't give in to the urge, there will be discomfort"... and as soon as you give in to the urge, the craving has gotten you to do what it "wanted" you to do, so it disappears and the discomfort that was associated with it disappears as well. So the craving just "proved" that you had to give in to the urge in order to avoid the discomfort from the negative valence... even though the discomfort was actually produced by the craving itself!

Whereas if you eliminated the craving to avoid this particular discomfort, then the discomfort from resisting the urge would also disappear. Note that this does not automatically mean that you would resist the urge: it just means that you'd have the option to, if you had some reason to do so. But falsifying the beliefs behind the craving is distinct from falsifying the beliefs that triggered the moral judgment system; you might still give in to the urge, if you believed it to be correct and justified. (This is part of my explanation for why it seems that you can reach high levels of enlightenment and see through the experience of the self, and still be a complete jerk towards others.)

Comment by kaj_sotala on Craving, suffering, and predictive processing (three characteristics series) · 2020-05-22T12:24:58.379Z · score: 2 (1 votes) · LW · GW

This reminds me of my discussion with johnswentworth, where I was the one arguing that model-free vs. model-based is a sliding scale. :)

So yes, it seems reasonable to me that these might be best understood as extreme ends of a spectrum... which was part of the reason why I copied that excerpt, as it included the concluding sentence of "‘Model-based’ and ‘model-free’ modes of valuation and response, if this is correct, simply name extremes along a single continuum and may appear in many mixtures and combinations determined by the task at hand" at the end. :)

Comment by kaj_sotala on A non-mystical explanation of "no-self" (three characteristics series) · 2020-05-22T09:55:27.898Z · score: 2 (1 votes) · LW · GW

(Sorry for the late response; I seem to have missed this comment earlier.)

'I'm not sure I understand. If you thought you were at the red dot rather than at the location in the world it marks, wouldn't that be analogous to thinking you are the feeling of tension, rather than to thinking you are at the location that feeling indicates?

Hmm, is there a difference? In that if you think that you are the feeling of tension, then logically you are also at the location of the tension.

I tried the exercise. I didn't know what you expected, but my idea of "noticing myself looking" is a model, so I found something like seeing myself staring at the thing from a third-person perspective. I think I could reproduce your result, but I'm writing this the day after, and now that Im no longer tired I have to create the tension on purpose. [...]

There is also a sense in which you are looking at the world from behind your eyes. Your visual image is a projection with the focal point behind your eyes. If you try the same exercise with holding something in your hand and feeling it rather than looking at something, how does that work out? I tried to do "the same thing" I did to reproduce the tension behind the eyes, and the sensation was just below my skin. I dont know if that's the "right" answer, but if it is, the fact that it's not in the head might suggest the previous result is an artifact.

Yes, subtle differences in how these kinds of exercises are framed produce different kinds of results. Noticing that is part of the point - if you examine one kind of experience, you may notice your brain telling you that you are in one place; if you examine another kind of experience, you may notice your brain telling you that you are in another place. Sometimes the "you" may be a feeling of tension, sometimes a feeling under your skin, sometimes a visual image. These kinds of inconsistencies suggest that a part of the experience of the self, is actually an interpretation that is constructed on the fly, rather than being fundamental in the sense that intuition might otherwise suggest.

(If you have the experience of seeing yourself staring at the thing from a third-person perspective, then a question that might be interesting to investigate is "where are you looking at the third-person image from?". Not trying to fit the answer into the model that I have explained here, nor going into any intellectual mode of analysis, but just paying attention to the experience and what the answer to that question might feel like...)

There are two quotes after that. The first seems congurent with what you said, but the second sounds like identifying with all the contents of consiousness rather than with the field they are in (or is that distinction not real?).

Good catch! I think it's basically the same, despite sounding different; I briefly say a few words about that at the end of a later post.

This is what I understood "identifying with the field of conciousness" to mean, is that right? I think I can do that, but it seems it's not compatible with goal-directed action, which would require its own self-markers as described.

It's possible to get into states where you have this to at least some extent, but there's also some goal-directed action going on; and you are identifying with a process which is observing that goal-directed action, rather than getting pulled into it.

That said, I don't want to say anything about what is "supposed" to happen, because that easily creates craving to experience the thing that's supposed to happen, and then craving warps the experience to make you see what it thinks that the thing will look like, which may not be the same thing. (See the next post about craving.) It's often better to not have very strong expectations, and just keep investigating what seems to happen when you do different things...