Posts

[link] If something seems unusually hard for you, see if you're missing a minor insight 2021-05-05T10:23:26.046Z
Beliefs as emotional strategies 2021-04-09T14:28:16.590Z
Open loops in fiction 2021-03-14T08:50:03.948Z
The three existing ways of explaining the three characteristics of existence 2021-03-07T18:20:24.298Z
Multimodal Neurons in Artificial Neural Networks 2021-03-05T09:01:53.996Z
Different kinds of language proficiency 2021-02-26T18:20:04.342Z
[Fiction] Lena (MMAcevedo) 2021-02-23T19:46:34.637Z
What's your best alternate history utopia? 2021-02-22T08:17:23.774Z
Internet Encyclopedia of Philosophy on Ethics of Artificial Intelligence 2021-02-20T13:54:05.162Z
Bedtime reminiscences 2021-02-19T11:50:05.271Z
Unwitting cult leaders 2021-02-11T11:10:04.504Z
[link] The AI Girlfriend Seducing China’s Lonely Men 2020-12-14T20:18:15.115Z
Are index funds still a good investment? 2020-12-02T21:31:40.413Z
Snyder-Beattie, Sandberg, Drexler & Bonsall (2020): The Timing of Evolutionary Transitions Suggests Intelligent Life Is Rare 2020-11-24T10:36:40.843Z
Retrospective: November 10-day virtual meditation retreat 2020-11-23T15:00:07.011Z
Memory reconsolidation for self-affection 2020-10-27T10:10:04.884Z
Group debugging guidelines & thoughts 2020-10-19T11:02:32.883Z
Things are allowed to be good and bad at the same time 2020-10-17T08:00:06.742Z
The Felt Sense: What, Why and How 2020-10-05T15:57:50.545Z
Public transmit metta 2020-10-04T11:40:03.879Z
Attention to snakes not fear of snakes: evolution encoding environmental knowledge in peripheral systems 2020-10-02T11:50:05.327Z
AI Advantages [Gems from the Wiki] 2020-09-22T22:44:36.671Z
The Haters Gonna Hate Fallacy 2020-09-22T12:20:06.050Z
(Humor) AI Alignment Critical Failure Table 2020-08-31T19:51:18.266Z
nostalgebraist: Recursive Goodhart's Law 2020-08-26T11:07:46.690Z
Collection of GPT-3 results 2020-07-18T20:04:50.027Z
Are there good ways to find expert reviews of popular science books? 2020-06-09T14:54:23.102Z
Three characteristics: impermanence 2020-06-05T07:48:02.098Z
On the construction of the self 2020-05-29T13:04:30.071Z
From self to craving (three characteristics series) 2020-05-22T12:16:42.697Z
Craving, suffering, and predictive processing (three characteristics series) 2020-05-15T13:21:50.666Z
A non-mystical explanation of "no-self" (three characteristics series) 2020-05-08T10:37:06.591Z
A non-mystical explanation of insight meditation and the three characteristics of existence: introduction and preamble 2020-05-05T19:09:44.484Z
Stanford Encyclopedia of Philosophy on AI ethics and superintelligence 2020-05-02T07:35:36.997Z
Healing vs. exercise analogies for emotional work 2020-01-27T19:10:01.477Z
The two-layer model of human values, and problems with synthesizing preferences 2020-01-24T15:17:33.638Z
Under what circumstances is "don't look at existing research" good advice? 2019-12-13T13:59:52.889Z
A mechanistic model of meditation 2019-11-06T21:37:03.819Z
On Internal Family Systems and multi-agent minds: a reply to PJ Eby 2019-10-29T14:56:19.590Z
Book summary: Unlocking the Emotional Brain 2019-10-08T19:11:23.578Z
System 2 as working-memory augmented System 1 reasoning 2019-09-25T08:39:08.011Z
Subagents, trauma and rationality 2019-08-14T13:14:46.838Z
Subagents, neural Turing machines, thought selection, and blindspots 2019-08-06T21:15:24.400Z
On pointless waiting 2019-06-10T08:58:56.018Z
Integrating disagreeing subagents 2019-05-14T14:06:55.632Z
Subagents, akrasia, and coherence in humans 2019-03-25T14:24:18.095Z
Subagents, introspective awareness, and blending 2019-03-02T12:53:47.282Z
Building up to an Internal Family Systems model 2019-01-26T12:25:11.162Z
Book Summary: Consciousness and the Brain 2019-01-16T14:43:59.202Z
Sequence introduction: non-agent and multiagent models of mind 2019-01-07T14:12:30.297Z

Comments

Comment by Kaj_Sotala on Taboo "Outside View" · 2021-06-19T17:43:44.363Z · LW · GW

Eliezer also wrote an interesting comment on the EA Forum crosspost of this article, copying it here for convenience:

I worriedly predict that anyone who followed your advice here would just switch to describing whatever they're doing as "reference class forecasting" since this captures the key dynamic that makes describing what they're doing as "outside viewing" appealing: namely, they get to pick a choice of "reference class" whose samples yield the answer they want, claim that their point is in the reference class, and then claiming that what they're doing is what superforecasters do and what Philip Tetlock told them to do and super epistemically virtuous and anyone who argues with them gets all the burden of proof and is probably a bad person but we get to virtuously listen to them and then reject them for having used the "inside view".

My own take:  Rule One of invoking "the outside view" or "reference class forecasting" is that if a point is more dissimilar to examples in your choice of "reference class" than the examples in the "reference class" are dissimilar to each other, what you're doing is "analogy", not "outside viewing".

All those experimental results on people doing well by using the outside view are results on people drawing a new sample from the same bag as previous samples.  Not "arguably the same bag" or "well it's the same bag if you look at this way", really actually the same bag: how late you'll be getting Christmas presents this year, based on how late you were in previous years.  Superforecasters doing well by extrapolating are extrapolating a time-series over 20 years, which was a straight line over those 20 years, to another 5 years out along the same line with the same error bars, and then using that as the baseline for further adjustments with due epistemic humility about how sometimes straight lines just get interrupted some year.  Not by them picking a class of 5 "relevant" historical events that all had the same outcome, and arguing that some 6th historical event goes in the same class and will have that same outcome.

Comment by Kaj_Sotala on Which rationalists faced significant side-effects from COVID-19 vaccination? · 2021-06-14T13:13:12.725Z · LW · GW

One Pfizer with no side-effects.

Comment by Kaj_Sotala on Against intelligence · 2021-06-09T16:30:29.664Z · LW · GW

But power seems to be very unrelated to intelligence.

On the level of individuals, perhaps. But one argument is that the more relevant question is that of species-level comparisons; if you need to understand people, to know them, befriend them, network with them, get them to like you, etc., then a human brain may be able to do it, but while a mice or dog brain might manage to do some of it, it's not going to get to a position of real power that way.

Eliezer making an argument on why one should explicitly not think of "intelligence" as corresponding to conceptual intelligence, but rather to "the thing that makes humans different from other animals":

General intelligence is a between-species difference, a complex adaptation, and a human universal found in all known cultures. There may as yet be no academic consensus on intelligence, but there is no doubt about the existence, or the power, of the thing-to-be-explained. There is something about humans that let us set our footprints on the Moon.

But the word “intelligence” commonly evokes pictures of the starving professor with an IQ of 160 and the billionaire CEO with an IQ of merely 120. Indeed there are differences of individual ability apart from “book smarts” which contribute to relative success in the human world: enthusiasm, social skills, education, musical talent, rationality. Note that each factor I listed is cognitive. Social skills reside in the brain, not the liver. And jokes aside, you will not find many CEOs, nor yet professors of academia, who are chimpanzees. You will not find many acclaimed rationalists, nor artists, nor poets, nor leaders, nor engineers, nor skilled networkers, nor martial artists, nor musical composers who are mice. Intelligence is the foundation of human power, the strength that fuels our other arts.

The danger of confusing general intelligence with g-factor is that it leads to tremendously underestimating the potential impact of Artificial Intelligence. (This applies to underestimating potential good impacts, as well as potential bad impacts.) Even the phrase “transhuman AI” or “artificial superintelligence” may still evoke images of booksmarts-in-a-box: an AI that’s really good at cognitive tasks stereotypically associated with “intelligence,” like chess or abstract mathematics. But not superhumanly persuasive; or far better than humans at predicting and manipulating human social situations; or inhumanly clever in formulating long-term strategies. So instead of Einstein, should we think of, say, the 19th-century political and diplomatic genius Otto von Bismarck? But that’s only the mirror version of the error. The entire range from village idiot to Einstein, or from village idiot to Bismarck, fits into a small dot on the range from amoeba to human.

If the word “intelligence” evokes Einstein instead of humans, then it may sound sensible to say that intelligence is no match for a gun, as if guns had grown on trees. It may sound sensible to say that intelligence is no match for money, as if mice used money. Human beings didn’t start out with major assets in claws, teeth, armor, or any of the other advantages that were the daily currency of other species. If you had looked at humans from the perspective of the rest of the ecosphere, there was no hint that the squishy things would eventually clothe themselves in armored tanks. We invented the battleground on which we defeated lions and wolves. We did not match them claw for claw, tooth for tooth; we had our own ideas about what mattered. Such is the power of creativity.

Comment by Kaj_Sotala on Rationalists should meet Integral Theory · 2021-06-05T14:17:58.484Z · LW · GW

I'm not familiar with Integral Theory, but I read an earlier book by Wilber that arguably also qualifies as "philosophy of life". I found it to contain some stuff that felt very valuable and some stuff that felt like obvious nonsense.

It strikes me that he was approaching the topics in a way that might be considered somewhat analogous to a study of cognitive biases - in that even if you do actually have a good theoretical understanding of biases that other people can learn from, it doesn't necessarily mean that you're any good at being less biased yourself. Or possibly you have managed to debias yourself with regard to some biases, but you keep getting blindsided by some trickier ones, even if you understand them in theory

This seems to me like a general issue with all these kinds of things, whether it's about cognitive bias or therapy or philosophy of life. You only ever see your mind from the inside, and simply knowing about how it works will (usually) not change how it actually works, so you can have a fantastic understanding of minds in general and manage to fix most of your issues and still fail to apply your skills to one gaping blindspot that's obvious to everyone around you. Or conversely, you can be a massive failure as a human being and have a million things you haven't addressed at all, but still be able to produce some valuable insights.

That said, I do agree that the more red flags there are around a person, the more cautious one should be - both in terms of epistemics (more risk of absorbing bad ideas) and due to general consequentialist reasons (if someone supports known abusers, then endorsing them may indirectly lend support to abuse). 

Comment by Kaj_Sotala on A non-mystical explanation of insight meditation and the three characteristics of existence: introduction and preamble · 2021-06-04T15:22:29.454Z · LW · GW

As someone who was very young at the time, I liked the idea of becoming "enlightened" and "letting go of my ego." I believed I could learn to use my time and energy for the benefit of other people and put away my 'selfish' desires to help myself, and even thought this was desirable. This backfired as I became a people-pleaser, and still find it hard to put my needs ahead of other peoples to this day.

I can't put this fully at the feet of my lone and ill advised forays into meditation, but it's only much later I learned the idea that in order to let go of something, you have to have it first. I don't think I had fully developed my ego at the point I started learning to "let it go" and healthy formation of identity is a crucial step to a happy life I think.

Great observation. I've experienced something similar (using meditative practices in an attempt to suppress my own needs and desires in a way that was ultimately detrimental). I also don't think it was really caused by meditation; rather it was an emotional wound (or a form of craving) masquerading as a noble intention. 

I don't recall hearing the "in order to let go of something, you have to have it first" line before, but I love it. You could say that I've been working to develop my ego recently, for a similar reason - wanting to get to a point where my needs are actually met rather than actively denying them.

Comment by Kaj_Sotala on Rationalists should meet Integral Theory · 2021-06-04T12:44:13.533Z · LW · GW

Agreed, I think this post would be much strengthened if it would include some kind of a summary of Integral Theory's main claims and some brief discussion of why Elo thinks they're correct.

Comment by Kaj_Sotala on TEAM: a dramatically improved form of therapy · 2021-06-03T17:05:27.568Z · LW · GW

If the therapist has to put down a probability on the patient having found the therapist empathic the therapist will be faster at learning when he's perceived as empathic by their patients then if the therapist just sees the numbers. 

I wonder how accurate these kinds of answers are going to be. At one point my self-improvement group was doing peer coaching sessions that involved giving your coach feedback at the end. I don't remember our exact questions, but questions about the coach's perceived empathy definitely sound like the kind of thing that could have been on the list.

I remember that when I'd been coached, I felt significantly averse to giving the person-who'd-just-done-their-best-to-help-me any critical feedback, especially on a trait such as empathy that people often interpret as reflecting on them as a person. I'd imagine that the status differential between a client and a therapist could easily make this worse, particularly in the case of clients who are specifically looking for help on something like poor self-esteem or excess people-pleasing. (Might not be a problem with patients who are there for being too disagreeable, though!)

Comment by Kaj_Sotala on TEAM: a dramatically improved form of therapy · 2021-06-03T16:50:15.315Z · LW · GW

Thanks, that's useful. I'd heard of some other reconsolidation-fans read Burns's new book and also highlight the "what's good about this" aspect of it as CBT "also coming around" to the "positive purpose" idea. So then when I thought I saw it in this post as well, I assumed that to be correct. Especially since that would have helped explain why TEAM is so effective.

Though interestingly this makes me somewhat more interested in TEAM, since it's obviously doing something different from what I already know, rather than just confirming my previous prejudices without adding new information. :-)

Comment by Kaj_Sotala on Power dynamics as a blind spot or blurry spot in our collective world-modeling, especially around AI · 2021-06-02T17:00:50.835Z · LW · GW

Oops, yeah. Edited.

Comment by Kaj_Sotala on Power dynamics as a blind spot or blurry spot in our collective world-modeling, especially around AI · 2021-06-02T15:56:21.142Z · LW · GW

In this post, I wish to share an opposing concern: that the EA and rationality communities have become systematically biased to ignore multi/multi dynamics, and power dynamics more generally.  

The EA and rationality communities tend to lean very strongly towards mistake rather than conflict theory. A topic that I've had in my mind for a while, but haven't gotten around writing a full post about, is of both of them looking like emotional defense strategies. 

It looks to me like Scott's post is pointing towards, not actually different theories, but one's underlying cognitive-emotional disposition or behavioral strategy towards outgroups. Do you go with the disposition towards empathizing and assuming that others are basically good people that you can reason with, or with the disposition towards banding together with your allies and defending against a potential threat?

And at least in the extremes, both of them look like they have elements of an emotional defense. Mistake doesn't want to deal with the issue that some people you just can't reason with no matter how good your intentions, so it ignores that and attempts to solve all problems by dialogue and reasoning. (Also, many Mistake Theorists are just bad at dealing with conflict in general.) Conflict doesn't want to deal with the issue that often people who hurt you have understandable reasons for doing so and that they are often hurting too, so it ignores that and attempts to solve all problems by conflict.

If this model is true, then it also suggests that Mistake Theorists should also be systematically biased against the possibility of things like power dynamics being genuinely significant. If power dynamics are genuinely significant, then you might have to resolve things by conflict no matter how much you invest in dialogue and understanding, which is the exact scenario that Mistake is trying to desperately avoid.

Comment by Kaj_Sotala on TEAM: a dramatically improved form of therapy · 2021-06-01T17:01:06.651Z · LW · GW

Hmm, so do you mean that TEAM does not actually assume issues to necessarily have a positive function, the idea that they might have is just one way of overcoming resistance?

Comment by Kaj_Sotala on Why don't long running conversations happen on LessWrong? · 2021-05-31T18:08:03.317Z · LW · GW

There are a few things that are tagged as "the great conversations on LessWrong" in my mind, and those are specifically ones that took the form of posts-as-responses. Two specific examples that I'm thinking of would be

  • Wei Dai's The Nature of Offense, which was a response to three earlier posts by Alicorn, orthonormal and Eliezer (posts which had in turn been responding to each other), and showed how each of them was a special case of a so-far unrecognized general principle of what offense is.
  • Morendil and my Red Paperclip Theory of Status; this was a post that Morendil and I co-authored after I had proposed a definition of status in the comments of a post that Morendil had made that was in turn responding to a number of other posts (mine one of them) about "what is status" on LW. I'm no doubt a little biased in considering this one of the great successes, but it felt pretty significant in that to me it felt like it did to the concept of status what Wei Dai's post did to the concept of offense: pulled together all the threads of the conflicting theories that'd been proposed so far to provide an overall synthesis and definition that the main participants in the discussion (in this case me and Morendil; not everyone seems to have found the post's model equally useful) agreed to have resolved it.

I would also want to be able to nominate Eliezer and Robin's FOOM debate, but while that one is certainly long and has them engaging each other, ultimately it didn't seem to bring their views substantially closer to each other - much unlike the two other examples I mentioned.

Comment by Kaj_Sotala on TEAM: a dramatically improved form of therapy · 2021-05-31T14:43:48.919Z · LW · GW

This sounds similar to memory reconsolidation -based therapies, which assume that any emotional issue that you have exists because that issue actually serves some purpose, and in fact your "problem" represents a solution for some other problem that you have had before. By acknowledging the positive purpose behind the issue, you can find a way to keep the purpose while changing the strategy.

I haven't listened to any of the episodes, though, so I'd be curious to hear whether you think that's talking about the same thing or something subtly different?

Comment by Kaj_Sotala on A Review and Summary of the Landmark Forum · 2021-05-28T15:43:39.962Z · LW · GW

I'm not familiar with Landmark, but the description of how they deal with narratives reminds me of therapy and memory reconsolidation; much of this sounds a lot like making unconscious beliefs and interpretations explicit so that they can then be disproven.

According to Landmark, the answer is simple, you just do (“all this time you thought you were trapped inside, but the door wasn’t even locked”). They illustrate this with the story of monkeys being trapped by putting a banana in a cage just big enough for them to put their hands through. As it goes, when the monkey tries to grab the banana, it finds its hand trapped as the hole isn’t big enough to pull it out. The monkey could escape, but it’s unwilling to let go of the banana. However, we could also interpret them as operating under the theory that if the understanding and realisation is strong enough and lands deep enough then it creates a shift automatically.

Unlocking the Emotional Brain notes that while making unconscious narratives explicit and conscious isn't always enough to disprove them, there are many cases where it is, because once they are explicit it is easier for the brain to notice how they contradict other things that it also believes. That would be in line with this kind of a theory.

Comment by Kaj_Sotala on Cortés, Pizarro, and Afonso as Precedents for Takeover · 2021-05-28T14:33:35.179Z · LW · GW

However, I don't think this is the whole explanation. The technological advantage of the conquistadors was not overwhelming.

With regard to the Americas at least, I just happened to read this article by a professional military historian, who characterizes the Native American military technology as being "thousands of years behind their Old World agrarian counterparts", which sounds like the advantage was actually rather overwhelming.

There is a massive amount of literature to explain what is sometimes called ‘the Great Divergence‘ (a term I am going to use here as valuable shorthand) between Europe and the rest of the world between 1500 and 1800. Of all of this, most readers are likely only to be familiar with one work, J. Diamond’s Guns, Germs and Steel (1997), which is unfortunate because Diamond’s model of geographic determinism is actually not terribly well regarded in the debate (although, to be fair, it is still better than some of the truly trash nationalistic nonsense that gets produced on this topic). Diamond asks the Great Divergence question with perhaps the least interesting framing: “Why Europe and not the New World?” and so we might as well get that question out of the way first.

I am well aware that when EU4 was released, this particular question – and generally the relative power of New World societies as compared to Old World societies – was a point of ferocious debate among fans (particularly on Paradox’s own forums). What makes this actually a less central question (though still an important one) is that the answer is wildly overdetermined. That is to say, any of these causes – the germs, the steel (through less the guns; Diamond’s attention is on the wrong developments there), but also horses, ocean-going ships, and dense, cohesive, disciplined military formations would have been enough in isolation to give almost any complex agrarian Old-World society military advantages which were likely to prove overwhelming in the event. The ‘killer technologies’ that made the conquest of the New World possible were (apart from the ships) old technologies in much of Afroeurasia; a Roman legion or a Han Chinese army of some fifteen centuries earlier would have had many of the same advantages had they been able to surmount the logistical problem of actually getting there. In the face of the vast shear in military technology (though often not in other technologies) which put Native American armies thousands of years behind their Old World agrarian counterparts, it is hard not to conclude that whatever Afroeurasian society was the first to resolve the logistical barriers to putting an army in the New World was also very likely to conquer it.

(On these points, see J.F. Guilmartin, “The Cutting Edge: An Analysis of the Spanish Invasion and Overthrow of the Inca Empire, 1532-1539,” in Transatlantic Encounters: European and Andeans in the Sixteenth Century, eds. K. J. Andrien and R. Adorno (1991) and W.E. Lee, “The Military Revolution of Native North America: Firearms, Forts and Politics” in Empires and Indigenes: Intercultural Alliance, Imperial Expansion and Warfare in the Early Modern World, eds. W.E. Lee (2011). Both provide a good sense of the scale of the ‘technological shear’ between old world and new world armies and in particular that the technologies which were transformative were often not new things like guns, but very old things, like pikes, horses and metal axes.)

With regard to the Indian Ocean, he writes:

the Portuguese cartaz-system (c. 1500-c. 1700) [was] the main way that the Portuguese and later European powers wrested control over trade in the Indian Ocean; it only worked because Portuguese warships were functionally unbeatable by anything else afloat in the region due to differences in local styles of shipbuilding).

Comment by Kaj_Sotala on The Homunculus Problem · 2021-05-28T12:09:45.783Z · LW · GW

I like the point that homunculus language is necessary for describing the experience of visual illusions, but I don't understand this part:

Even if there was a little person, the argument does not describe my subjective experience, because I can still see the shadow! I experience the shadowed area as darker than the unshadowed area. So the homunculus story doesn't actually fit what I see at all!

How does seeing the shadow contradict the explanation? Isn't the explanation meant to say that the appearance of the shadow is the result of "the brain adjusting colors for us", in that the brain infers the existence of a shadow and then adjusts the image to incorporate that?

Comment by Kaj_Sotala on supposedlyfun's Shortform · 2021-05-25T16:05:47.011Z · LW · GW

I'm of the type to get easily addicted to notifications, and daily has felt rare enough for me to not trigger any reaction.

Comment by Kaj_Sotala on We should probably buy ADA? · 2021-05-25T16:02:57.444Z · LW · GW

A brief summary of "what is Cardano" would be helpful (I'd never heard of it before this post).

Comment by Kaj_Sotala on What is the strongest argument you know for antirealism? · 2021-05-13T13:17:38.592Z · LW · GW

It seems very hard for me to imagine how one could create a procedure that wasn't biased towards a particular value system. E.g. Stuart Armstrong has written about how humans can be assigned any values whatsoever - you have to decide what parts of their behavior are because of genuine preferences and what parts are because of irrationality, and what values that implies. And the way you decide what's correct behavior and what's irrationality seems like the kind of a choice that will depend on your own values. Even something like "this seems like the simplest way of assigning preferences" presupposes that it is valuable to pick a procedure based on its simplicity - though the post argues that even simplicity would fail to distinguish between several alternative ways of assigning preferences.

Of course, just because we can't be truly unbiased doesn't mean we couldn't be less biased, so maybe something like "pick the simplest system that produces sensible agents, distinguishing between ties at random" could arguably be the least biased alternative. But human values seem quite complex; if there was some simple and unbiased solution that would produce convergent values to all AIs that implemented it, it might certainly have something in common with what we call values, but that's not a very high bar. There's a sense in which all the bacteria share the same goal, "making more (surviving) copies of yourself is the only thing that matters", and I'd expect the convergent value system to end up as being something like that. That has some resemblance to human values, since many humans also care about having offspring, but not very much.

Comment by Kaj_Sotala on What is the strongest argument you know for antirealism? · 2021-05-12T14:52:26.344Z · LW · GW

My position is something like "I haven't yet seen anyone compellingly both define and argue for moral realism, so until then the whole notion seems confused to me".

It is unclear to me what it would even mean for a moral claim to actually be objectively true or false. At the same time, there are many evolutionary and game-theoretical reasons for why various moral claims would feel objectively true or false to human minds, and that seems sufficient for explaining why many people have an intuition of moral realism being true. I have also personally found some of my moral beliefs changing as a result of psychological work - see the second example here - which makes me further inclined to believe that moral beliefs are all psychological (and thus subjective, as I understand the term).

So my argument is simply that there doesn't seem to be any reason for me to believe in moral realism, somewhat analogous to how there doesn't seem to be any reason for me to believe in a supernatural God.

Comment by Kaj_Sotala on What are your favorite examples of adults in and around this community publicly changing their minds? · 2021-05-10T16:53:55.941Z · LW · GW

As far as I understand, "I changed my mind about the claims in the paper" isn't usually considered a reason to withdraw. Withdrawal is something like an attempt to retract the fact that you ever made a claim in the first place, and reserved for things like outright fraud or very serious mistakes in data collection that invalidate the whole analysis. 

Comment by Kaj_Sotala on Spoiler-Free Review: Monster Train · 2021-05-08T07:28:35.402Z · LW · GW

25 hours played, this game has been growing on me as well. Though I too would like it if there was more room for creative improvisation, as in Slay the Spire, rather than strict optimization. A lot of the units have just outright fun concepts, and it would be nice if mixing and matching them more freely would make gameplay sense. 

At 220 hours played and with Covenant 25 wins on 14 of the base game's 20 possible class pairings, I don't feel this way anymore. Improvisation and relentless optimization aren't opposed; the need for relentless optimization just requires me to improvise better. It just required getting familiar enough with the mechanics and cards to be able to do that.

Comment by Kaj_Sotala on Covid 5/6: Vaccine Patent Suspension · 2021-05-07T18:45:40.266Z · LW · GW

Is it disagreeing with the OP's original premise of "lifting the patents doesn't do anything", or with my inference of "if lifting the patents doesn't do anything, then it won't change future drug company behavior either"? I'm not sure how I'd tell, but I'd presume the more near-term premise of "lifting the patents does(n't) do anything" would have a bigger effect on immediate stock prices.

Comment by Kaj_Sotala on Covid 5/6: Vaccine Patent Suspension · 2021-05-07T13:41:52.015Z · LW · GW

That still makes the OP sound rather extreme, though:

If you do not think school’s primary nature is ‘child prison’ and/or that those running it are pro-children, then you have new data your model needs to somehow explain. 

"The people running the school system genuinely think that school is for education and learning, and are happy to have found an option that would allow children to keep learning even on days when they otherwise wouldn't have" seems like a perfectly reasonable explanation, even if one disagrees with that reasoning.

Comment by Kaj_Sotala on Covid 5/6: Vaccine Patent Suspension · 2021-05-07T13:35:52.843Z · LW · GW

The Biden administration’s latest strategy for the pandemic is to suspend the vaccine patents without compensation. Our life expectancies are lower than they were last week. [...]

Many people have this idea that all the knowledge and skill required to produce the vaccines lies in the patents. Once you lift the patents, lots of other companies can go start producing vaccines. Except, that’s not actually true because

  1. The vaccines require technical expertise not included in the patents, which is expensive and slow to transfer, and which would also transfer valuable knowledge that can be used for other R&D and other production and thus which the vaccine producers are not going to transfer without compensation.
  2. Moderna explicitly already said they wouldn’t enforce the patents, and no one really expected the others to either.

Read that second one again, if it’s new to you. The greedy capitalists whose rights you took away without compensation were already voluntarily giving those rights away. If there was already clearly no intent to enforce the patents, what good does lifting those patents do?

It feels odd to me to simultaneously argue that patents were unimportant and unenforced anyway so this will produce no benefit, and that the decision to suspend patents will hurt the drug companies so much that in the future they have less of an incentive to invest in drug development?

Seems like "this will kill people" is premised on the assumption that the next time something like COVID happens, drug companies will remember what happened last so will be less incentivized to invest. But if the thing that they remember the next time is "governments did this purely symbolic gesture that didn't really affect our profits in any way but made their voters happier", then it would seem like they should have no particular reason to act any different?

Comment by Kaj_Sotala on My Journey to the Dark Side · 2021-05-07T07:28:53.178Z · LW · GW

Similar. I also pretty strongly think that e.g. many of the core beliefs driving our behavior have nothing to do with their stated reasons and are actually social strategies. But the whole hemisphere thing, as well as the vehemence that Ziz has around her moral views, don't seem compelling to me. They give me a strong vibe of themselves being derived from some unacknowledged emotional strategy she has going on, rather than reflecting any kind of real truth.

Comment by Kaj_Sotala on All is fair in love and war, on Zero-sum games in life · 2021-05-06T08:12:33.019Z · LW · GW

More specifically, interpersonal interaction has both a dominance dimension ("of status, dominance, power, ambitiousness, assertiveness, or control") and a warmth dimension ("of agreeableness, compassion, nurturant, solidarity, friendliness, warmth, affiliation or love"). Dominance is zero-sum, but warmth is not.

Cultures also vary in how much they emphasize the dominance and warmth dimensions. In more "status-flat" cultures (such as the Nordic countries), social conventions tend to de-emphasize status differences, making relative status less important and letting the warmth dimension matter more.

It seems interesting to me that I feel like I mostly encounter arguments such as "status is zero-sum so we can't ever make everyone happy" expressed by people from non-Nordic countries. The notion always seemed unintuitive to me, and I don't think that the reason is just "Kaj personally pays less attention to dominance status" since I do feel pretty sensitive to it. Rather, it feels like a significant part of it is Finnish culture just not caring about dominance status that much, relative to warmth, making it hard for me to see why the zero-sumness of status should necessarily be a significant problem.

Comment by Kaj_Sotala on Why I Work on Ads · 2021-05-06T07:18:54.721Z · LW · GW

From the perspective of users, I think the internet would be essentially unusable unless you subscribed to a few standard services, which would then have harmful levels of leverage. 

I wonder about that: before third-party services started popping up, internet service providers and nonprofits used to offer more services that are now offered by third parties. E.g. your ISP used to give you an e-mail account and website space, and services such as Usenet and IRC functioned in a decentralized fashion, with servers being hosted by universities, ISPs and others. That model won't work for everything, but it doesn't seem like too much of a stretch to imagine services such as social media and search shifting to a more decentralized model if advertising was banned. (Decentralized social media networks such as Diaspora already exist; I'm under the impression that the main reason they're not used more is that network effects create too much lock-in to existing, more centralized services.)

Comment by Kaj_Sotala on [link] If something seems unusually hard for you, see if you're missing a minor insight · 2021-05-06T05:47:41.217Z · LW · GW

Thanks for pointing that out, edited to make it a little clearer.

Comment by Kaj_Sotala on Your Dog is Even Smarter Than You Think · 2021-05-05T13:11:01.683Z · LW · GW

From reading the discussion, I get the impression that some of the commenters are writing from a position of "the prior is against significant dog intelligence, and the evidence here could have alternative explanations, so I'm skeptical". That is, the people feel that it would be quite surprising if dogs really were this intelligent, so establishing it requires some pretty compelling evidence.

At the same time, my own feeling is more like "there's no strong prior against significant dog intelligence, and the evidence here could have alternative explanations, so I'm being tentatively open to this being real". As in, even if you hadn't shown me these videos, dogs being approximately as intelligent as described would have seemed like an entirely plausible possibility to me.

If there are any people who feel like my first paragraph does describe them, I'd be curious to hear why they're coming into this discussion with such a strong prior against dog intelligence.

If I had to articulate the reasons for my own "seems plausible" prior, they'd be something like:

  • a general vague sense of animal research tending to generally show that animals are smarter than people often think
  • some of the animals in books like "Don't Shoot the Dog" sounding relatively smart (e.g. vaguely recall the author mentioning that training often speeds up once the animal figures out that it's being trained, since then it can explicitly try to figure out what the trainer is trying to reward it for)

I once videotaped a beautiful Arabian mare who was being clicker-trained to prick her ears on command, so as to look alert in the show ring. She clearly knew that a click meant a handful of grain. She clearly knew her actions made her trainer click. And she knew it had something to do with her ears. But what? Holding her head erect, she rotated her ears individually: one forward, one back; then the reverse; then she flopped both ears to the sides like a rabbit, something I didn't know a horse could do on purpose. Finally, both ears went forward at once. Click! Aha! She had it straight from then on. It was charming, but it was also sad: We don't usually ask horses to think or to be inventive, and they seem to like to do it.

  • even relatively simple AI systems exhibiting surprisingly intelligent behavior (GPT-3), suggesting that there isn't necessarily a sharp distinction between human and less-than-human intelligence
Comment by Kaj_Sotala on Why I Work on Ads · 2021-05-05T10:57:00.406Z · LW · GW

The type of ad that merely informs you of an existence of a product is possible in theory, and maybe existed in 19th century, but I was born too late for that.

I have purchased clothes, plush animals, books, and games because of online advertisements that told me about their existence; I would have been unaware of the products in question if not for the ads. (I have also generally been happy with the products that I got; one of the clothes that I ordered is probably my favorite piece of clothing.)

Comment by Kaj_Sotala on Announcing The Inside View Podcast · 2021-05-05T05:48:12.855Z · LW · GW

CastingWords at least used to be accurate (the one time I used them, I don't recall the transcript having had any flaws).

Comment by Kaj_Sotala on Your Dog is Even Smarter Than You Think · 2021-05-03T10:21:43.822Z · LW · GW

So the danger is over-interpreting their output. Was this sentence intended or just random babbling? Does the dog understand the word differently from what it means in English? E.g. "bye" seems to become a verb meaning "leaving", "love you" is used for affection but obviously doesn't reflect deep understanding of the human concept of love. 

Interestingly, there's an argument that human infants also learn language by their parents over-interpreting their input, with the infants then adopting those interpretations as true. So one could argue that even if over-interpretation happens with dogs, that only makes it a process similar to human language learning, with the parent/child and owner/dog creating a shared language game.

Our first type of example comes from our own data concerning Zulu infants of between three and four months of age interacting with their mothers, and suggests an answer to this question. [...]

As noted above, there are times when a caregiver will want an infant to fall silent, or in isiZulu to 'thula'. Zulu children are traditionally expected to be less socially active than contemporary Western children, to initiate fewer interactions, and, crucially, to show a respectful attitude towards adults. An early manifestation of this is in behaviours where a mother attempts to make an infant keep quiet, sometimes saying 'thula' ('quiet'), 'njega' ('no'), while simultaneously gesturing, moving towards or away from the infant, and reacting to details of the infants own behaviour (see Cowley et al., in press).

At these times the mother regularly leans forward, so that more of the infants visual field is taken up by her face and palms. New vocalisations, and movements or re-orientations of gaze by the infant, are often 'nipped in the bud' by dominating vocalisations (sometimes showing prosodic properties indicative of disapproval, comforting, attention and/or arousal towards the mother herself) from the mother, sometimes accompanied by increasingly emphatic hand-waving, and even closer crowding of the infants visual field. [...]

With high regularity, and within relatively little time, the particular infant often does 'thula', at which point it is generally rewarded with smiling, gentle touching, and other comforting.

At this stage there is no reason to believe that the infant knows what 'thula' or 'njega' means, or even that it could reliably re-identify the words, let alone produce or contemplate them, so it is extremely unlikely that the word-based aspects of maternal utterance-activity provide labels for the infant. We are considering infants before the stage linguists call 'babbling', let alone recognisable speech production. It is not even necessary to suppose that it 'knows' that it is supposed to be quiet when behaved at in the ways we have just described. We know that the mother wants the child to be quiet, that this expresses itself in behaviour by the mother, and that the infant comes to be quiet.

If we examine the mothers behaviour, though, we can make sense of it. She ensures that it is difficult for the infant to attend to anything else by crowding its visual field. She rejects active or new behaviours on its part by cutting off its vocalisations and movements with dominating signals of her own. She largely restricts approval signals, including relaxing the crowding, and reducing the magnitude of her gesturing, as well as expressing comfort through vocalisation, facial signalling and touch, to moments when the infant begins to quieten down. Its not particularly surprising, then, that it does quieten down.

The mothers behaviour includes salient, repeated, features which are apt for learning. Her patterns of hand gesturing, for example, could at the outset be iconic of the whole episode including her behaviour and the infants becoming quiet, but, when repetition allows the gesture to be individuated and recognised in its own right, go on to become an indexical cue that quietness should follow. The infants responses then become indexical for the mother of the degree to which the child is co-operative, well-behaved, or, more plainly, 'good'. Caregiver descriptions of infant behaviour at these times, manifest either in their explicit vocalisations to the child, including references to being 'good', or references to possible disciplinary sanctions such as 'kuza baba manje' ('wheres your father now?') or, in interviews following the videotaping, show that infant behaviour even at this early age is being classified in line with culturally specific expectations of good and bad behaviour. And a crucial part of what makes for a 'good' child is responding in ways sensitive to what caregiver behaviour is actually about, strikingly in controlling episodes such as the one just described, which make possible the earliest ascriptions of 'obedience', 'cooperativeness' and so forth.

These ascriptions are over-interpretations. They are, though, necessary overinterpretations, in so far as they motivate caregivers to imbue their own behaviour with regularities manifest regularities in their own behaviour which are then available as structure in the interactional environment for (learning by) the infant. A further episode from our data, in this case concerning a child of around four months, illustrates this point about over-interpretation. In it an infant repeatedly vocalises in ways which to its mother, at least, are suggestive of its saying 'up'. Each time she says 'up'?, or 'you want to go up'? and after a few repetitions she lifts the child. Prior to the lifting, there is little evidence that the child actually wants to be lifted, or that it has its attention focussed on anything in particular, except perhaps its own experiments in vocal control. When it is lifted, though, it beams widely. Whatever it did want, if anything, it is now, we suggest, one step closer to figuring out how to behave in ways that lead to its being lifted up.

Still on the subject of lifting, consider the common gesture made around the eighth month by infants who want to be picked up (that is, who subsequently smile or otherwise show approval when they are picked up following such a gesture): a simultaneous raising, or flapping, of both arms (see Lock, 1991). This gesture is not simply copied from common adult behaviours. In the terms we are using here it is partly iconic, in virtue of being a common posture of infants while they are in fact being held up, and partly indexical, in virtue of being able to stand on its own as an indicator of 'being up', as well as being symbolically interpretable as an invitation to lift, or a request to be lifted. Such gestures are, importantly, serviceable label candidates, in virtue of being amenable to disembedding from behaviour, and eventually coming under deliberate control. An infant need not want to be lifted the first few times it makes such a gesture, it has only to be able to notice that the gesture tends to be followed by liftings.

If and when such learning takes place, it does so in the affectively charged environment we have briefly described. We want to bring discussion of the current example to a close by suggesting a way in which these interactions should be regarded as a further example of how minds can be extended through action. Clark and Chalmers suggestion is that paradigmatically mental states and processes can be realised by structures and resources external to the brain. The world beyond the skull of any individual includes, of course, the skulls and brains of others. If active externalism motivates the recognition of a cognitive prosthesis such as a filofax as part of what realises a mind, then the embodied brain of another can also play that role. Here, then, is our suggestion: that at times interacting caregiver-infant dyads are neither one individual nor two, but somewhere in between. At the risk of sounding sensational and un-PC at the same time, infant brains can be temporarily colonised by caregivers so as to accelerate learning processes. [...]

The instances of indexical learning we describe also permit the beginning of a kind of 'semiotic arms race' between infants and caregivers. Once an infant has learned, for example, that the arms-up gesture can lead to being lifted, it is possible for 'requests' (that is, behaviours taken as requests by others, no matter how they are to the infant) to be lifted to be acted on, or to be refused. Prior to the construction and learning of the indexical relationship, this was impossible––a parent would lift a child when the parent wanted to, or thought it would serve some end. Once it has been learned, 'requests' can be differentially responded to, depending on their situation in patterns of interaction extending through time. Personal and cultural contingencies about infants and parents will codetermine what patterns are formed, and whether, for example, requested lifting is more likely after relatively quick acquiescence to silencing behaviour, or less likely in the period following failure to attend to objects or events in which a caregiver attempted to arouse interest.

Comment by Kaj_Sotala on Death by Red Tape · 2021-05-02T18:39:19.372Z · LW · GW

In the Zones of Thought universe, there is a cycle of civilization: civilizations rise from stone-age technology, gradually accumulating more technology, until they reach the peak of technological possibility. At that point, the only way they can improve society is by over-optimizing it for typical cases, removing slack. Once society has removed its slack, it's just a matter of time until unforeseen events force the system slightly outside of its safe parameters. This sets off a chain reaction: like dominoes falling, the failure of one subsystem causes the failure of another and another. This catastrophe either kills everyone on the planet, or sets things so far back that society has to start from scratch.

Reminds me of this:

In 1988, Joseph Tainter wrote a chilling book called The Collapse of Complex Societies. Tainter looked at several societies that gradually arrived at a level of remarkable sophistication then suddenly collapsed: the Romans, the Lowlands Maya, the inhabitants of Chaco canyon. Every one of those groups had rich traditions, complex social structures, advanced technology, but despite their sophistication, they collapsed, impoverishing and scattering their citizens and leaving little but future archeological sites as evidence of previous greatness. Tainter asked himself whether there was some explanation common to these sudden dissolutions.

The answer he arrived at was that they hadn’t collapsed despite their cultural sophistication, they’d collapsed because of it. Subject to violent compression, Tainter’s story goes like this: a group of people, through a combination of social organization and environmental luck, finds itself with a surplus of resources. Managing this surplus makes society more complex—agriculture rewards mathematical skill, granaries require new forms of construction, and so on.

Early on, the marginal value of this complexity is positive—each additional bit of complexity more than pays for itself in improved output—but over time, the law of diminishing returns reduces the marginal value, until it disappears completely. At this point, any additional complexity is pure cost.

Tainter’s thesis is that when society’s elite members add one layer of bureaucracy or demand one tribute too many, they end up extracting all the value from their environment it is possible to extract and then some.

The ‘and them some’ is what causes the trouble. Complex societies collapse because, when some stress comes, those societies have become too inflexible to respond. In retrospect, this can seem mystifying. Why didn’t these societies just re-tool in less complex ways? The answer Tainter gives is the simplest one: When societies fail to respond to reduced circumstances through orderly downsizing, it isn’t because they don’t want to, it’s because they can’t.

In such systems, there is no way to make things a little bit simpler – the whole edifice becomes a huge, interlocking system not readily amenable to change. Tainter doesn’t regard the sudden decoherence of these societies as either a tragedy or a mistake—”[U]nder a situation of declining marginal returns collapse may be the most appropriate response”, to use his pitiless phrase. Furthermore, even when moderate adjustments could be made, they tend to be resisted, because any simplification discomfits elites.

When the value of complexity turns negative, a society plagued by an inability to react remains as complex as ever, right up to the moment where it becomes suddenly and dramatically simpler, which is to say right up to the moment of collapse. Collapse is simply the last remaining method of simplification.

Comment by Kaj_Sotala on [Linkpost] Teaching Paradox, Europa Univeralis IV, Part I: State of Play · 2021-05-02T11:58:31.333Z · LW · GW

The article also references/discusses Seeing Like a State, which has been somewhat popular in rationalist circles after the SSC book review.

Comment by Kaj_Sotala on GPT-3 Gems · 2021-04-21T16:20:18.093Z · LW · GW

Apparently we'll be able to build lots of drones.

Comment by Kaj_Sotala on You Can Now Embed Flashcard Quizzes in Your LessWrong posts! · 2021-04-19T18:31:32.047Z · LW · GW
  • the general advice is that using cards written by others is much less useful than cards you write yourself

Seems like using cards by others is still better than not using any cards at all?

Comment by Kaj_Sotala on [Lecture Club] Awakening from the Meaning Crisis · 2021-04-19T18:21:42.040Z · LW · GW

So Vervaeke is familiar with dreams, and expects his audience to be familiar with dreams. Your sense of how much things cohere can be hacked! I realized this as the result of direct experience many years ago, as presumably have most people, and so any claim of states of consciousness that are more in touch with reality than the default state of consciousness, rather than less in touch with it, has a high bar of evidence to clear. The default presumption should be "how are you sure it isn't just hacking your sense of how much things cohere?"

Doesn't detract from your point, but I find it interesting that you interpreted dreams as evidence in this direction rather than the opposite. After all, when we are awake, we know we are awake, and correctly feel that our reality is more coherent and true than dreams are. The opposite isn't true: if we realize we're dreaming, we typically also realize that the content isn't true; we don't end up thinking that dreams are actually more true that reality is. Rather, finding dreams to be coherent requires us to not realize we're dreaming.

So feels like someone could just as easily have generalized this into saying "if there's an alternate state that on an examination feels more true than ordinary wakefulness does, then it's likely to actually be more true, in the same way as ordinary wakefulness both feels and is more true than dreams are".

Comment by Kaj_Sotala on What Multipolar Failure Looks Like, and Robust Agent-Agnostic Processes (RAAPs) · 2021-04-19T12:51:51.338Z · LW · GW

Thankfully, there have already been some successes in agent-agnostic thinking about AI x-risk

Also Sotala 2018 mentions the possibility of control over society gradually shifting over to a mutually trading collective of AIs (p. 323-324) as one "takeoff" route, as well as discussing various economic and competitive pressures to shift control over to AI systems and the possibility of a “race to the bottom of human control” where state or business actors [compete] to reduce human control and [increase] the autonomy of their AI systems to obtain an edge over their competitors (p. 326-328).

Sotala & Yampolskiy 2015 (p. 18) previously argued that:

In general, any broad domain involving high stakes, adversarial decision making and a need to act rapidly is likely to become increasingly dominated by autonomous systems. The extent to which the systems will need general intelligence will depend on the domain, but domains such as corporate management, fraud detection and warfare could plausibly make use of all the intelligence they can get. If oneʼs opponents in the domain are also using increasingly autonomous AI/AGI, there will be an arms race where one might have little choice but to give increasing amounts of control to AI/AGI systems.

Comment by Kaj_Sotala on Beliefs as emotional strategies · 2021-04-11T09:23:44.183Z · LW · GW

Of course, we can't know for sure. It could be that the interventions actually worked by a different method than they seemed to.

But consider e.g. the first story. Here was a person who started out entirely convinced that the belief in free will was an intrinsically hardwired need that they had. It had had a significant impact on their entire life, to the point of making them suicidally depressed when they couldn't believe it. I had a theory of how the mind works which made a different prediction, and I only needed to briefly suggest it for them to surface compatible evidence without me needing to make any more leading comments. After that, I only needed to suggest a single intervention which my model predicted would cause a change, and it did, causing a long-term and profound change in the other person.

Because I do expect it to be a permanent change rather than just a short-term effect. Of course, the first two examples are both from this year - I didn't ask Sampo when exactly his example happened - so in principle it's still possible that these will reverse themselves. But that's not my general experience with these things - rather, these interventions tend to produce permanent and lasting change. The longest-term effect I have personal data for is from June 2017; this follow-up from December 2018 still remains a good summary of what that intervention ended up fixing in the long term. (As noted in that follow-up, it's still possible for some issues to come back in a subtler form, or for some of the issues to also have other causes; but that's distinct from the original issue coming back in its original strength.)

So it's possible that my model is mistaken about the exact causality - but that by treating the model as if it was true, you're still able to cause lasting and deep changes in people's psychology.  If my model is wrong, then we need another model that would explain the same observations. Currently I think that the kinds of models that I've outlined would explain those observations pretty well while being theoretically plausible, but I'm certainly open to alternative ones. 

I don't think that e.g. just "hearing the right emotional story can produce relief" is a very good alternative theory. I've certainly also had experience of superficial emotional stories that sounded compelling for a little while and whose effect then faded out, but over time I've learned that a heuristic of "do these effects last for longer than a month" is pretty good for telling those apart from the ones that have a real effect. The permanent ones may also have an effect on things you didn't even realize were related beforehand - e.g. the person in the first example analyzing the things that they realized about it in retrospect - whereas in my experience, the short-term ones mostly just include effects that are obviously and directly derivable from the story. 

So some compelling stories seem to produce relatively minor short-term effects while other interventions cause much broader and longer-lasting ones, and just the hypothesis of "emotional stories can be compelling" doesn't explain why some emotional stories work better than others. Nor would it have predicted that suggesting the specific intervention that I offered would have been particularly useful. 

All of that said, I do admit that the third story has more interacting pieces and that the overall evidence for that one is weaker. We can only be relatively sure that telling the client to imagine a different kind of mother was the final piece in resolving the issue; it's possible that the other inferences about the mother's beliefs are incorrect. I still wanted to include it, in the spirit of learning soft skills, because I think that many beliefs that affect our behavior aren't nice and clear-cut ones where you can just isolate a single key belief and be relatively sure of what happened because you can observe the immediate effects. Rather there's much more behavior that's embedded in an interacting web of beliefs like I outlined there. Even if the details of that particular story were off, enough of it resonates in my inner simulator that I'm pretty sure that something like that story could be true and often is true. But for that one I can't offer a more convincing argument than "load it up in your own inner sim and see whether it resonates".

Comment by Kaj_Sotala on Testing The Natural Abstraction Hypothesis: Project Intro · 2021-04-07T17:17:44.191Z · LW · GW

Oh cool! I put some effort into pursuing a very similar idea earlier:

I'll start this post by discussing a closely related hypothesis: that given a specific learning or reasoning task and a certain kind of data, there is an optimal way to organize the data that will naturally emerge. If this were the case, then AI and human reasoning might naturally tend to learn the same kinds of concepts, even if they were using very different mechanisms.

but wasn't sure of how exactly to test it or work on it so I didn't get very far.

One idea that I had for testing it was rather different; make use of brain imaging research that seems able to map shared concepts between humans, and see whether that methodology could be used to also compare human-AI concepts:

A particularly fascinating experiment of this type is that of Shinkareva et al. (2011), who showed their test subjects both the written words for different tools and dwellings, and, separately, line-drawing images of the same tools and dwellings. A machine-learning classifier was both trained on image-evoked activity and made to predict word-evoked activity and vice versa, and achieved a high accuracy on category classification for both tasks. Even more interestingly, the representations seemed to be similar between subjects. Training the classifier on the word representations of all but one participant, and then having it classify the image representation of the left-out participant, also achieved a reliable (p<0.05) category classification for 8 out of 12 participants. This suggests a relatively similar concept space between humans of a similar background.

We can now hypothesize some ways of testing the similarity of the AI's concept space with that of humans. Possibly the most interesting one might be to develop a translation between a human's and an AI's internal representations of concepts. Take a human's neural activation when they're thinking of some concept, and then take the AI's internal activation when it is thinking of the same concept, and plot them in a shared space similar to the English-Mandarin translation. To what extent do the two concept geometries have similar shapes, allowing one to take a human's neural activation of the word "cat" to find the AI's internal representation of the word "cat"? To the extent that this is possible, one could probably establish that the two share highly similar concept systems.

One could also try to more explicitly optimize for such a similarity. For instance, one could train the AI to make predictions of different concepts, with the additional constraint that its internal representation must be such that a machine-learning classifier trained on a human's neural representations will correctly identify concept-clusters within the AI. This might force internal similarities on the representation beyond the ones that would already be formed from similarities in the data.

The farthest that I got with my general approach was "Defining Human Values for Value Learners". It felt (and still feels) to me like concepts are quite task-specific: two people in the same environment will develop very different concepts depending on the job that they need to perform...  or even depending on the tools that they have available. The spatial concepts of sailors practicing traditional Polynesian navigation are sufficiently different from those of modern sailors that the "traditionalists" have extreme difficulty understanding what the kinds of birds-eye-view maps we're used to are even representing - and vice versa; Western anthropologists had considerable difficulties figuring out what exactly it was that the traditional navigation methods were even talking about. 

(E.g. the traditional way of navigating from one island to another involves imagining a third "reference" island and tracking its location relative to the stars as the journey proceeds. Some anthropologists thought that this third island was meant as an "emergency island" to escape to in case of unforeseen trouble, an interpretation challenged by the fact that the reference island may sometimes be completely imagined, so obviously not suitable as a backup port. Chapter 2 of Hutchins 1995 has a detailed discussion of the way that different tools for performing navigation affect one's conceptual representations, including the difficulties both the anthropologists and the traditional navigators had in trying to understand each other due to having incompatible concepts.)

Another example are legal concepts; e.g. American law traditionally held that a landowner did not only control his land but also everything above it, to “an indefinite extent, upwards”. Upon the invention of this airplane, this raised the question: could landowners forbid airplanes from flying over their land, or was the ownership of the land limited to some specific height, above which the landowners had no control?

Eventually, the law was altered so that landowners couldn't forbid airplanes from flying over their land. Intuitively, one might think that this decision was made because the redefined concept did not substantially weaken the position of landowners, while allowing for entirely new possibilities for travel. In that case, we can think that our concept for landownership existed for the purpose of some vaguely-defined task (enabling the things that are commonly associated with owning land); when technology developed in a way that the existing concept started interfering with another task we value (fast travel), the concept came to be redefined so as to enable both tasks most efficiently.

This seemed to suggest an interplay between concepts and values; our values are to some extent defined in terms of our concepts, but our values and the tools that we have available for furthering our values also affect that how we define our concepts. This line of thought led me to think that that interaction must be rooted in what was evolutionarily beneficial:

... evolution selects for agents which best maximize their fitness, while agents cannot directly optimize for their own fitness as they are unaware of it. Agents can however have a reward function that rewards behaviors which increase the fitness of the agents. The optimal reward function is one which maximizes (in expectation) the fitness of any agents having it. Holding the intelligence of the agents constant, the closer an agent’s reward function is to the optimal reward function, the higher their fitness will be. Evolution should thus be expected to select for reward functions that are closest to the optimal reward function. In other words, organisms should be expected to receive rewards for carrying out tasks which have been evolutionarily adaptive in the past. [...]

We should expect an evolutionarily successful organism to develop concepts that abstract over situations that are similar with regards to receiving a reward from the optimal reward function. Suppose that a certain action in state s1 gives the organism a reward, and that there are also states s2–s5 in which taking some specific action causes the organism to end up in s1. Then we should expect the organism to develop a common concept for being in the states s2–s5, and we should expect that concept to be “more similar” to the concept of being in state s1 than to the concept of being in some state that was many actions away.

In other words, we have some set of innate values that our brain is trying to optimize for; if concepts are task-specific, then this suggests that the kinds of concepts that will be natural to us are those which are beneficial for achieving our innate values given our current (social, physical and technological) environment. E.g. for a child, the concepts of "a child" and "an adult" will seem very natural, because there are quite a few things that an adult can do for furthering or hindering the child's goals that fellow children can't do. (And a specific subset of all adults named "mom and dad" is typically even more relevant for a particular child than any other adults are, making this an even more natural concept.)

That in turn seems to suggest that in order to see what concepts will be natural for humans, we need to look at fields such as psychology and neuroscience in order to figure out what our innate values are and how the interplay of innate and acquired values develops over time. I've had some hope that some of my later work on the structure and functioning of the mind would be relevant for that purpose.

Comment by Kaj_Sotala on [deleted post] 2021-04-06T15:07:00.277Z

Yeah, subagents is the general idea of modeling the mind in terms of independent agents, but IFS is a more specific theory of what kinds of subagents there are. E.g. my sequence has a post about understanding System 1 and System 2 in terms of subagents, while IFS doesn't really have anything to say about that.

Comment by Kaj_Sotala on What Do We Know About The Consciousness, Anyway? · 2021-04-01T18:14:15.291Z · LW · GW

But this idea - self-consciousness is a model trained to predict other such models and generalizing to itself - seems both extremely obvious (in retrospective) and as mentioned before, with one small exception I can’t remember ever hearing or reading about it.

The idea feels familiar enough that I didn't feel surprised to see you suggest it, but I'm not sure where exactly I might have first encountered it. Learning to be conscious seems like a somewhat similar model, at least:

Consciousness remains a formidable challenge. Different theories of consciousness have proposed vastly different mechanisms to account for phenomenal experience. Here, appealing to aspects of global workspace theory, higher-order theories, social theories, and predictive processing, we introduce a novel framework: the self-organizing metarepresentational account (SOMA), in which consciousness is viewed as something that the brain learns to do. By this account, the brain continuously and unconsciously learns to redescribe its own activity to itself, so developing systems of metarepresentations that qualify target first-order representations. Thus, experiences only occur in experiencers that have learned to know they possess certain first-order states and that have learned to care more about certain states than about others. In this sense, consciousness is the brain’s (unconscious, embodied, enactive, nonconceptual) theory about itself.

As does maybe this paper [edit: apparently it's written by the person who wrote the "Rethinking Consciousness" book]:

One possible explanation of consciousness, proposed here, is that it is a construct of the social perceptual machinery. Humans have specialized neuronal machinery that allows us to be socially intelligent. The primary role for this machinery is to construct models of other people’s minds thereby gaining some ability to predict the behavior of other individuals. In the present hypothesis, awareness is a perceptual reconstruction of attentional state; and the machinery that computes information about other people’s awareness is the same machinery that computes information about our own awareness. The present article brings together a variety of lines of evidence including experiments on the neural basis of social perception, on hemispatial neglect, on the out-of-body experience, on mirror neurons, and on the mechanisms of decision-making, to explore the possibility that awareness is a construct of the social machinery in the brain.

I'm also somewhat reminded Thomas Metzinger's stuff about consciousness being a "self-model" (though it tends to be a bit of a pain to figure out what the heck exactly he's saying; I didn't even try doing more than skimming that page, and wouldn't recommend that to others, either), Dennett's notion of the self as a narrative center of gravity, and this LW comment.

Comment by Kaj_Sotala on How do we prepare for final crunch time? · 2021-04-01T10:49:10.637Z · LW · GW

Does any military use meditation as part of its training? 

. Yes, e.g.

This [2019] winter, Army infantry soldiers at Schofield Barracks in Hawaii began using mindfulness to improve shooting skills — for instance, focusing on when to pull the trigger amid chaos to avoid unnecessary civilian harm.

The British Royal Navy has given mindfulness training to officers, and military leaders are rolling it out in the Army and Royal Air Force for some officers and enlisted soldiers. The New Zealand Defence Force recently adopted the technique, and military forces of the Netherlands are considering the idea, too.

This week, NATO plans to hold a two-day symposium in Berlin to discuss the evidence behind the use of mindfulness in the military.

A small but growing group of military officials support the techniques to heal trauma-stressed veterans, make command decisions and help soldiers in chaotic battles.

“I was asked recently if my soldiers call me General Moonbeam,” said Maj. Gen. Piatt, who was director of operations for the Army and now commands its 10th Mountain Division. “There’s a stereotype this makes you soft. No, it brings you on point.”

The approach, he said, is based on the work of Amishi Jha, an associate professor of psychology at the University of Miami. She is the senior author of a paper published in December about the training’s effectiveness among members of a special operations unit.

The paper, in the journal Progress in Brain Research, reported that the troops who went through a monthlong training regimen that included daily practice in mindful breathing and focus techniques were better able to discern key information under chaotic circumstances and experienced increases in working memory function. The soldiers also reported making fewer cognitive errors than service members who did not use mindfulness.

The findings, which build on previous research showing improvements among soldiers and professional football players trained in mindfulness, are significant in part because members of the special forces are already selected for their ability to focus. The fact that even they saw improvement speaks to the power of the training, Dr. Jha said. [...]

Mr. Boughton has thought about whether mindfulness is anathema to conflict. “The purists would say that mindfulness was never developed for war purpose,” he said.

What he means is that mindfulness is often associated with peacefulness. But, he added, the idea is to be as faithful to compassionate and humane ideals as possible given the realities of the job.

Maj. Gen. Piatt underscored that point, describing one delicate diplomatic mission in Iraq that involved meeting with a local tribal leader. Before the session, he said, he meditated in front of a palm tree, and found himself extremely focused when the delicate conversation took place shortly thereafter.

“I was not taking notes. I remember every word she was saying. I wasn’t forming a response, just listening,” he said. When the tribal leader finished, he said, “I talked back to her about every single point, had to concede on some. I remember the expression on her face: This is someone we can work with.”

Comment by Kaj_Sotala on Rationalism before the Sequences · 2021-03-31T05:45:48.642Z · LW · GW

I think this comment would make for a good top-level post almost as it is.

Comment by Kaj_Sotala on Rationalism before the Sequences · 2021-03-30T19:48:56.252Z · LW · GW

I was slightly surprised, mostly because I had the expectation that if you've known about LW for a while, then I would have thought that you'd end up contributing either early or not at all. Curious what caused it to happen in 2021 in particular.

Comment by Kaj_Sotala on Rationalism before the Sequences · 2021-03-30T19:36:58.845Z · LW · GW

I also quite liked both the Jargon File (which I found before or around the same time as LW) and Dancing With the Gods (which I found through LW).

Comment by Kaj_Sotala on What Happens To Your Brain When You Write? · 2021-03-30T18:55:49.835Z · LW · GW

Similarly, if I'm writing something original, then if I'm typing I can type relatively close to the speed of my thought - it feels like my words are only somewhat trailing behind the shape of what I'm about to say. But if I'm writing by hand, there's more "lag", with it feeling like it takes much longer for my writing to catch up to the thought.

On the other hand, this feels like it has positive consequences; the words taking longer to write out, means that I also spend more time processing their content, and maybe the writing is a little better as a result. But having to wait for so long also feels frustrating, which is why I mostly don't do it.

Comment by Kaj_Sotala on Voting-like mechanisms which address size of preferences? · 2021-03-30T18:15:36.293Z · LW · GW

What kind of election do these governments use?

Mostly, I think, voting systems designed to ensure that parties get a share of seats that's proportional to their number of votes ("party-list proportional representation" is what Wikipedia calls it). E.g. the D'Hondt method seems pretty popular (and is used in Finland as well as several other countries).

As for whether it's actually better overall - well, I grew up with it and am used to it so I prefer it over something that would produce a two-party system. ;) But I don't have any very strong facts to present over which system is actually best.

Comment by Kaj_Sotala on Toward A Bayesian Theory Of Willpower · 2021-03-29T08:00:54.401Z · LW · GW

Actually upon further thought, I disagree with Scott's premise that this case allows for a meaningful distinction between "instinctual" and "intellectual" processes, so I guess I agree with you.