Rob B's Shortform Feed

Shared with permission, a google doc exchange confirming Eliezer still finds the arguments for alignment optimism, slower takeoffs, etc. unconvincing:

Daniel Filan: I feel like a bunch of people have shifted a bunch in the type of AI x-risk that worries them (representative phrase is "from Yudkowsky/Bostrom to What Failure Looks Like part 2 part 1") and I still don't totally get why.

Eliezer Yudkowsky: My bitter take:  I tried cutting back on talking to do research; and so people talked a bunch about a different scenario that was nicer to think about, and ended up with their thoughts staying there, because that's what happens if nobody else is arguing them out of it.

That is: this social-space's thought processes are not robust enough against mildly adversarial noise, that trying a bunch of different arguments for something relatively nicer to believe, won't Goodhart up a plausible-to-the-social-space argument for the thing that's nicer to believe.  If you talk people out of one error, somebody else searches around in the space of plausible arguments and finds a new error.  I wasn't fighting a mistaken argument for why AI niceness isn't too intractable and takeoffs won't be too fast; I was fighting an endless generator of those arguments.  If I could have taught people to find the counterarguments themselves, that would have been progress.  I did try that.  It didn't work because the counterargument-generator is one level of abstraction higher, and has to be operated and circumstantially adapted too precisely for the social-space to be argued into it using words.

You can sometimes argue people into beliefs.  It is much harder to argue them into skills.  The negation of Robin Hanson's rosier AI scenario was a belief.  Negating an endless stream of rosy scenarios is a skill.

Caveat: this was a private reply I saw and wanted to share (so people know EY's basic epistemic state, and therefore probably the state of other MIRI leadership). This wasn't an attempt to write an adequate public response to any of the public arguments put forward for alignment optimism or non-fast takeoff, etc., and isn't meant to be a replacement for public, detailed, object-level discussion. (Though I don't know when/if MIRI folks plan to produce a proper response, and if I expected such a response soonish I'd probably have just waited and posted that instead.)

Rolf Degen, summarizing part of Barbara Finlay's "The neuroscience of vision and pain":

Humans may have evolved to experience far greater pain, malaise and suffering than the rest of the animal kingdom, due to their intense sociality giving them a reasonable chance of receiving help.

From the paper:

Several years ago, we proposed the idea that pain, and sickness behaviour had become systematically increased in humans compared with our primate relatives, because human intense sociality allowed that we could ask for help and have a reasonable chance of receiving it. We called this hypothesis ‘the pain of altruism’ [68]. This idea derives from, but is a substantive extension of Wall’s account of the placebo response [43]. Starting from human childbirth as an example (but applying the idea to all kinds of trauma and illness), we hypothesized that labour pains are more painful in humans so that we might get help, an ‘obligatory midwifery’ which most other primates avoid and which improves survival in human childbirth substantially ([67]; see also [69]). Additionally, labour pains do not arise from tissue damage, but rather predict possible tissue damage and a considerable chance of death. Pain and the duration of recovery after trauma are extended, because humans may expect to be provisioned and protected during such periods. The vigour and duration of immune responses after infection, with attendant malaise, are also increased. Noisy expression of pain and malaise, coupled with an unusual responsivity to such requests, was thought to be an adaptation.

We noted that similar effects might have been established in domesticated animals and pets, and addressed issues of ‘honest signalling’ that this kind of petition for help raised. No implication that no other primate ever supplied or asked for help from any other was intended, nor any claim that animals do not feel pain. Rather, animals would experience pain to the degree it was functional, to escape trauma and minimize movement after trauma, insofar as possible.

Finlay's original article on the topic: "The pain of altruism".

Facebook comment I wrote in February, in response to the question 'Why might having beauty in the world matter?':

I assume you're asking about why it might be better for beautiful objects in the world to exist (even if no one experiences them), and not asking about why it might be better for experiences of beauty to exist.

[... S]ome reasons I think this:

1. If it cost me literally nothing, I feel like I'd rather there exist a planet that's beautiful, ornate, and complex than one that's dull and simple -- even if the planet can never be seen or visited by anyone, and has no other impact on anyone's life. This feels like a weak preference, but it helps get a foot in the door for beauty.

(The obvious counterargument here is that my brain might be bad at simulating the scenario where there's literally zero chance I'll ever interact with a thing; or I may be otherwise confused about my values.)

2. Another weak foot-in-the-door argument: People seem to value beauty, and some people claim to value it terminally. Since human value is complicated and messy and idiosyncratic (compare person-specific ASMR triggers or nostalgia triggers or culinary preferences) and terminal and instrumental values are easily altered and interchanged in our brain, our prior should be that at least some people really do have weird preferences like that at least some of the time.

(And if it's just a few other people who value beauty, and not me, I should still value it for the sake of altruism and cooperativeness.)

3. If morality isn't "special" -- if it's just one of many facets of human values, and isn't a particularly natural-kind-ish facet -- then it's likelier that a full understanding of human value would lead us to treat aesthetic and moral preferences as more coextensive, interconnected, and fuzzy. If I can value someone else's happiness inherently, without needing to experience or know about it myself, it then becomes harder to say why I can't value non-conscious states inherently; and "beauty" is an obvious candidate. My preferences aren't all about my own experiences, and they aren't simple, so it's not clear why aesthetic preferences should be an exception to this rule.

4. Similarly, if phenomenal consciousness is fuzzy or fake, then it becomes less likely that our preferences range only and exactly over subjective experiences (or their closest non-fake counterparts). Which removes the main reason to think unexperienced beauty doesn't matter to people.

Combining the latter two points, and the literature on emotions like disgust and purity which have both moral and non-moral aspects, it seems plausible that the extrapolated versions of preferences like "I don't like it when other sentient beings suffer" could turn out to have aesthetic aspects or interpretations like "I find it ugly for brain regions to have suffering-ish configurations".

Even if consciousness is fully a real thing, it seems as though a sufficiently deep reductive understanding of consciousness should lead us to understand and evaluate consciousness similarly whether we're thinking about it in intentional/psychologizing terms or just thinking about the physical structure of the corresponding brain state. We shouldn't be more outraged by a world-state under one description than under an equivalent description, ideally.

But then it seems less obvious that the brain states we care about should exactly correspond to the ones that are conscious, with no other brain states mattering; and aesthetic emotions are one of the main ways we relate to things we're treating as physical systems.

As a concrete example, maybe our ideal selves would find it inherently disgusting for a brain state that sort of almost looks conscious to go through the motions of being tortured, even when we aren't the least bit confused or uncertain about whether it's really conscious, just because our terminal values are associative and symbolic. I use this example because it's an especially easy one to understand from a morality- and consciousness-centered perspective, but I expect our ideal preferences about physical states to end up being very weird and complicated, and not to end up being all that much like our moral intuitions today.

Addendum: As always, this kind of thing is ridiculously speculative and not the kind of thing to put one's weight down on or try to "lock in" for civilization. But it can be useful to keep the range of options in view, so we have them in mind when we figure out how to test them later.

Collecting all of the quantitative AI predictions I know of MIRI leadership making on Arbital (let me know if I missed any):

• Aligning an AGI adds significant development time: Eliezer 95%
• Almost all real-world domains are rich: Eliezer 80%
• Complexity of value: Eliezer 97%, Nate 97%
• Distant superintelligences can coerce the most probable environment of your AI: Eliezer 66%
• Meta-rules for (narrow) value learning are still unsolved: Eliezer 95%
• Natural language understanding of "right" will yield normativity: Eliezer 10%
• Relevant powerful agents will be highly optimized: Eliezer 75%
• Some computations are people: Eliezer 99%, Nate 99%
• Sufficiently optimized agents appear coherent: Eliezer 85%

Some caveats:

• Arbital predictions range from 1% to 99%.
• I assume these are generally ~5 years old. Views may have shifted.
• By default, I assume that the standard caveats for probabilities like these apply: I treat these as off-the-cuff ass numbers [LW · GW] unless stated otherwise, products of 'thinking about the problem on and off for years and then querying my gut about what it expects to actually see', more so than of building Guesstimate models or trying to hard to make sure all the probabilities are perfectly coherent.

Inconsistencies are flags 'something is wrong here', but ass numbers are vague and unreliable enough that they're to be expected to some degree. Similarly, ass numbers are often unstable hour-to-hour and day-to-day.

Suppose most people think there's a shrew in the basement, and Richard Feynman thinks there's a beaver. If you're pretty sure it's not a shrew, two possible reactions include:

- 'Ah, the truth is probably somewhere in between these competing perspectives. So maybe it's an intermediate-sized rodent, like a squirrel.'

- 'Ah, Feynman has an absurdly good epistemic track record, and early data does indicate that the animal's probably bigger than a shrew. So I'll go with his guess and say it's probably a beaver.'

But a third possible response is:

- 'Ah, if Feynman's right, then a lot of people are massively underestimating the rodent's size. Feynman is a person too, and might be making the same error (just to a lesser degree); so my modal guess will be that it's something bigger than a beaver, like a capybara.'

In particular, you may want to go more extreme than Feynman if you think there's something systematically causing people to underestimate a quantity (e.g., a cognitive bias -- the person who speaks out first against a bias might still be affected by it, just to a lesser degree), or systematically causing people to make weaker claims than they really believe (e.g., maybe people don't want to sound extreme or out-of-step with the mainstream view).

From Twitter:

I am not sure I can write out the full AI x-risk scenario.

1. AI quickly becomes super clever

2. Alignment is hard, like getting your great x10 grandchildren to think you're a good person

3. The AI probably embarks on a big project which ignores us and accidentally kills us

Where am I wrong? Happy to be sent stuff to read.

I replied:

"1. AI quickly becomes super clever"

My AI risk model (which is not the same as everyone's) more specifically says:

1a. We'll eventually figure out how to make AI that's 'generally good at science' -- like how humans can do sciences that didn't exist when our brains evolved.

1b. AGI / STEM AI will have a large, fast, and discontinuous impact. Discontinuous because it's a new sort of intelligence (not just AlphaGo 2 or GPT-5); large and fast because STEM is powerful, plus humans suck at STEM and aren't cheap software that scales as you add hardware.

(Warning: argument is compressed for Twitter character count. There are other factors too, like recursive self-improvement.)

"2. Alignment is hard, like getting your great x10 grandchildren to think you're a good person"

I'd say it's hard like building a large/complex, novel software system that exhibits some strong robustness/security properties, on the first try, in the face of adverse optimization.

My recommended reading would be:

1. https://intelligence.org/2017/11/25/security-mindset-ordinary-paranoia/ 
2. https://intelligence.org/2017/11/26/security-mindset-and-the-logistic-success-curve/ 

Goal stability over time is part of the problem, but not the core problem. The core problem (for ML) is 'ML models are extremely opaque, and there's no way to robustly get any complex real-world goal into a sufficiently opaque system'. The goal isn't instilled in the first place.

"3. The AI probably embarks on a big project which ignores us and accidentally kills us"

Rather: which deliberately kills us because (1) we're made of atoms that can be used for the project, and (2) we're a threat. (E.g., we could build a rival superintelligence.)

Chana Messinger, replying to Brandon Bradford:

I find this very deep

"Easy to make everything a conspiracy when you don't know how anything works."

Everything literally is a conspiracy (in some nonstandard technical sense), and if you don't know how anything works, then it's a secret conspiracy.

How does water get to your faucet? How many people are responsible for your internet? What set of events had to transpire to make you late for work? How does one build a microwave?

Something about this points at how complicated everything is and how little we individually know about it.

From an April 2019 Facebook discussion:

Rob Bensinger: avacyn:

I think one strong argument in favor of eating meat is that beef cattle (esp. grass-fed) might have net positive lives. If this is true, then the utilitarian line is to 1) eat more beef to increase demand, 2) continue advocating for welfare reforms that will make cows' lives even more positive.

Beef cattle are different than e.g. factory farmed chicken in that they live a long time (around 3 years on average vs 6-7 weeks for broilers), and spend much of their lives grazing on stockers where they might have natural-ish lives.

Another argument in favor of eating beef is that it tends to lead to deforestation, which decreases total wild animal habitat, which one might think are worse than beef farms.

... I love how EA does veganism / animal welfare things. It's really good.

(From the comment section on https://forum.effectivealtruism.org/posts/TyLxMrByKuCmzZx6b/reasons-to-eat-meat [EA · GW])

[... Note that in posting this I'm not intending] to advocate for a specific intervention; it's more that it makes me happy to see thorough and outside-the-box reasoning from folks who are trying to help others, whether or not they have the same background views as me.

Jonathan Salter: Even if this line of reasoning might technically be correct in a narrow, first order effects type way, my intuition tells me that that sort of behaviour would lessen EAs credibility when pushing animal welfare messages, and that spreading general anti-speciesist norms and values to be more important in the long run. Just my two cents though.

Rob Bensinger: My model of what EA should be shooting for is that it should establish its reputation as

'that group of people that engages in wonkish analyses and debates of moral issues at great length, and then actually acts on the conclusions they reach'

'that group of people that does lots of cost-benefit analyses and is willing to consider really counter-intuitive concerns rather than rejecting unusual ideas out of hand'

'that group of people that seems to be super concerned about its actual impact and nailing down all the details, rather than being content with good PR or moral signaling'

I think that's the niche EA would occupy if it were going to have the biggest positive impact in the future. And given how diverse EA is and how many disagreements there already are, the ship may have already sailed on us being able to coordinate and converge on moral interventions without any public discussion of things like wild animal suffering.

This is similar to a respect in which my views have changed about whether EAs and rationalists should become vegan en masse. In the past, I've given arguments like [in Inhuman Altruism and Revenge of the Meat People]:

• A lot more EAs and rationalists should go vegan, because it really does seem like future generations will view 21st-century factory farming similar to how we view 19th-century slavery today. It would be great to be "ahead of the curve" for once, and to clearly show that we're not just 'unusually good on some moral questions' but actually morally exemplary in all the important ways that we can achieve.

I think this is really standing in for two different arguments:

First, a reputational argument saying 'veganism is an unusually clear signal that we're willing to take big, costly steps to do the right thing, and that we're not just armchair theorists or insular contrarians; so we should put more value on paying that signal in order to convince other people that we're really serious about this save-the-world, help-others, actually-act-based-on-the-abstract-arguments thing'.

Second, an ideal-advisor-style argument saying 'meat-eating is probably worse than it seems, because the analytic arguments strongly support veganism but social pressure and social intuitions don't back up those arguments, so we probably won't emotionally feel their full moral force'.

One objection I got to the first argument is that it seems like the marginal effort and attention of a lot of EAs could save a lot more lives if it were going to things that can have global effects, rather than small-scale personal effects. The reputational argument weighs against this, but there's a reputational argument going in the other way (I believe due to Katja Grace [update: Katja Grace wrote an in-depth response at the time, but this particular argument seems to be due to Paul Christiano and Oliver Habryka]):

'What makes EA's brand special and distinctive, and puts us in an unusual position to have an outsized impact on the world, is that we're the group that gets really finicky and wonkish about EV and puts its energy into the things that seem highest-EV for the world. Prioritizing personal dietary choices over other, better uses of our time, and especially doing so for reputational or signaling reasons, seem like it actively goes against that unique aspect of EA, which makes it very questionable as a PR venture in the first place.'

This still left me feeling, on a gut level, like the 'history will view us as participants in an atrocity' argument is a strong one -- not as a reputational argument, just as an actual argument (by historical analogy) that there's something morally wrong with participating in factory farming at all, even if we're (in other aspects of our life) trying to actively oppose factory farming or even-larger atrocities.

Since then, a few things have made me feel like the latter argument's force isn't so strong. First, I've updated some on the object level about the probability that different species are conscious and that different species in particular circumstances have net-negative lives (though I still think there's a high-enough-to-be-worth-massively-worrying-about probability that farmed chicken, beef, etc. are all causing immense amounts of suffering).

Second, I've realized that when I've done my 'what would future generations think?' ideal-advisor test in the past, I've actually been doing something weird. I'm taking 21st-century intuitions about which things matter most and are most emotionally salient, and projecting them forward to imagine a society where salience works the same way on the meta level, but the object-level social pressures/dynamics are a bit different. But it seems like that heuristic might have been the wrong one for past generations to use, if they wanted to make proper use of this ideal-advisor heuristic.

Jeremy Bentham's exemplary forward-thinking moral views, for example, seem like a thing you'd achieve by going 'imagine future generations that are just super reasonable and analytical about all these things, and view things as atrocities in proportion to what the strongest arguments say', rather than by drawing analogies to things that present-day people find especially atrocious about the past.

(People who have read Bentham: did Bentham ever use intuition pumps like either of these? If so, did either line of thinking seem like it actually played a role in how he reached his conclusions, as opposed to being arguments for persuading others?)

Imagine instead a future society that's most horrified, above all else, by failures of reasoning process like 'foreseeably allocating attention and effort to something other than the thing that looks highest-EV to you'. Imagine a visceral gut-level reaction to systematic decision-making errors (that foreseeably have very negative EV) even more severe than modernity's most negative gut-level reactions to world events. Those failures of reasoning process, after all, are much more directly in your control (and much more influencable by moral praise and condemnation) than action outcomes. That seems like a hypothetical that pushes in a pretty different direction in a lot of these cases. (And one that converges more with the obvious direct 'just do the best thing' argument, which doesn't need any defense.)

While your comment was clearly written in good faith, it seems to me like you're missing some context. You recommend that EY recommend that the detractors read books. EY doesn't just recommend people read books. He wrote the equivalent of like three books [? · GW] on the subjects relevant to this conversation in particular which he gives away for free. Also, most of the people in this conversation are already big into reading books.

It is my impression he also helped establish the Center for Applied Rationality, which has the explicit mission of training skills. (I'm not sure if he technically did but he was part of the community which did and he helped promote it in its early days.)

From an April 2019 Facebook discussion:

Rob Bensinger:

Julia Galef: Another one of your posts that has stayed with me is a post in which you were responding to someone's question -- I think the question was, “What are your favorite virtues?” And you described three. They were compassion for yourself; creating conditions where you'll learn the truth; and sovereignty. [...] Can you explain briefly what sovereignty means?

Kelsey Piper: Yeah, so I characterize sovereignty as the virtue of believing yourself qualified to reason about your life, and to reason about the world, and to act based on your understanding of it.

I think it is surprisingly common to feel fundamentally unqualified even to reason about what you like, what makes you happy, which of several activities in front of you you want to do, which of your priorities are really important to you.

I think a lot of people feel the need to answer those questions by asking society what the objectively correct answer is, or trying to understand which answer won't get them in trouble. And so I think it's just really important to learn to answer those questions with what you actually want and what you actually care about. [...]

Julia Galef: One insight that I had from reading your post in particular was that maybe a lot of debates over whether you should "trust your gut” are actually about sovereignty. [...]

Kelsey Piper: Yeah, I definitely think -- maybe replace “trust your gut” with --

Julia Galef: Consult?

Kelsey Piper: Yeah, check in with your gut. Treat your gut as some information.

Julia Galef: Yeah.

Kelsey Piper: And treat making your gut more informative as an important part of your growth as a person. [...] I’ve stewed over lots of hard questions. And I got a sense of when I've tended to be right, and when I tended to be wrong, and that informs my gut and the extent to which I feel able to trust it now.

Source: http://rationallyspeakingpodcast.org/show/rs-230-kelsey-piper-on-big-picture-journalism-covering-the-t.html

Based on https://theunitofcaring.tumblr.com/post/177842591031/what-are-your-favorite-virtues

Spencer Mulesky: Why is this good content? Im not getting it.

Rob Bensinger: That seems hard to summarize!

The "trust your gut" portion maybe obscures the thing I think is important, because it seems more banal and specific. The important thing I think is being pointed at with "sovereignty" is more general than just "notice how you feel about things, and hone your intuitions through experience", though that's certainly a core thing people need to do.

One way of pointing at the more basic thing I have in mind is: by default, humans are pretty bad at being honest with themselves and others; are pretty bad at thinking clearly; are pretty bad at expressing and resolving disagreements, as opposed to conformity/mimicry or unproductive brawls; are pretty bad at taking risks and trying new things; are pretty bad at attending to argument structure, as opposed to status/authority/respectability.

We can build habits and group norms that make it a lot easier to avoid those problems, and to catch ourselves when we slip up. But this generally requires that people see past abstractions like "what I should do" and "what's normal to do" and "what's correct to do" and be able to observe and articulate what concrete things are going on in their head. A common thing that blocks this is that people feel like there's something silly or illegitimate or un-objective about reporting what's really going on inside their heads, so they feel a need to grasp at fake reasons that sound more normal/objective/impartial. Giving EAs/rationalists/etc. more social credit for something like "sovereignty", and giving them language for articulating this ideal, is one way of trying to fight back against this epistemically and instrumentally bad set of norms and mental habits.

Spencer Mulesky: Thanks!

Rob Bensinger: It might also help to give some random examples (with interesting interconnections) where I've found this helpful.

• 'I'm in a longstanding relationship that's turned sour. But I feel like I can't just leave (or make other changes to my life) because I'm not having enough fun / my life isn't satisfying as many values as I'd like; I feel like I need to find something objectively Bad my partner has done, so that I can feel Justified and Legitimate in leaving.' People often feel like they're not "allowed" to take radical action to improve their lives, because of others' seeming claims on their life.
• A lot of the distinct issues raised on https://www.facebook.com/robbensinger/posts/10160749026995447, like Jessica Taylor's worry about using moral debt as a social lever to push people around. In my experience, this is not so dissimilar from the relationship case above; people think about their obligations in fuzzy ways that make it hard to see what they actually want and easy to get trapped by others' claims on their output.
• People feel like they're being looked down on or shamed or insufficiently socially rewarded/incentivized/respected for things about how they're trying to do EA. Examples might include 'starting risky projects', 'applying to EA jobs', 'applying to non-EA jobs', 'earning to give', 'not earning to give', 'producing ideas that aren't perfectly vetted or write-ups that aren't perfectly polished'. (See e.g. the comments on https://www.facebook.com/robbensinger/posts/10161249846505447; or for the latter point, https://www.lesswrong.com/posts/7YG9zknYE8di9PuFg/epistemic-tenure [LW · GW] and a bunch of other recent writings share a theme of 'highly productive intellectuals are feeling pressure to not say things publicly until they're super super confident of them').
• People feel like they can't (even in private conversation) acknowledge social status and esteem (https://www.facebook.com/robbensinger/posts/10160475363630447), respectability, or 'Ra'-type things (https://srconstantin.wordpress.com/2016/10/20/ra), or how those phenomena affects everyone's preferences or judgments.
• Modest epistemology in Inadequate Equilibria (https://equilibriabook.com/toc/)
• A lot of CFAR-style personal debugging I've done has depended on my ability to catch myself in the mental motion of punishing myself (or disregarding myself, etc.) on the timescale of 'less than a second'. And then stopping to change that response or analyze why that's happening, so I can drill down better on underlying dispositions I want to improve. Cf. https://wiki.lesswrong.com/wiki/Ugh_field and https://www.lesswrong.com/posts/5dhWhjfxn4tPfFQdi/physical-and-mental-behavior [LW · GW].

Twitter thread collecting examples of alignment research MIRI has said relatively positive things about.

Copied from some conversations on Twitter:

· · · · · · · · · · · · · · · 

Eric Rogstad: I think "illusionism" is a really misleading term. As far as I can tell, illusionists believe that consciousness is real, but has some diff properties than others believe.

It's like if you called Einstein an "illusionist" w.r.t. space or time.

See my comments here:

https://www.lesswrong.com/posts/biKchmLrkatdBbiH8/book-review-rethinking-consciousness [LW · GW] 

Rob Bensinger: I mostly disagree. It's possible to define a theory-neutral notion of 'consciousness', but I think it's just true that 'there's no such thing as subjective awareness / qualia / etc.', and I think this cuts real dang deep into the heart of what most people mean by consciousness.

Before the name illusionism caught on, I had to use the term 'eliminativism', but I had to do a lot of work to clarify that I'm not like old-school eliminativists who think consciousness is obviously or analytically fake. Glad to have a clearer term now.

I think people get caught up in knots about the hard problem of consciousness because they try to gesture at 'the fact that they have subjective awareness', without realizing they're gesturing at something that contains massive introspective misrepresentations / illusions.

Seeing that we have to throw out a key part of Chalmers' explanandum is an important insight for avoiding philosophical knots, even though it doesn't much help us build a positive account. That account matters, but epistemic spring cleaning matters too.

Eric Rogstad: 

'the fact that they have subjective awareness', without realizing they're gesturing at something that contains massive introspective misrepresentations / illusions

@robbensinger I don't see how this is so different from my relativity example.

The situation still seems to be that there is a real thing that we're pointing at with "subjective awareness", and also people have a lot of wrong beliefs about it.

I guess the question is whether consciousness is more like "space" or more like "phlogiston".

It's possible to define a theory-neutral notion of 'consciousness'

@robbensinger If we defined a theory-neutral notion of 'consciousness' (to refer to whatever the thing is that causes us to talk about our experiences), would you still want to describe yourself as an 'illusionist' w.r.t that theory?

Keith Frankish: Yes! In fact that's exactly what I do. (The claim isn't that consciousness itself is illusory, only that qualia are)

Rob Bensinger: Yeah, I'm happy to define 'consciousness' in theory-neutral terms and say it exists in some sense. 'Qualia are an illusion' or 'phenomenal consciousness is an illusion' is more precise anyway.

I don't care how we define 'consciousness'. The claims I care about are:

1. Illusionism is asserting something substantive and (even among physicalists) controversial.

2. Illusionism is genuinely denying the existence of something widely seen as real and crucial (and self-evident!).

3. On the object level: phenomenal consciousness is in fact an introspective illusion.

'Phenomenal consciousness isn't real', 'phenomenal consciousness is real and reducible', and 'phenomenal consciousness is real and fundamental' are three substantively different views.

It's apparently not true that 90% of startups fail. From Ben Kuhn:

Hot take: the outside view is overrated.

(“Outside view” = e.g. asking “what % of startups succeed?” and assuming that’s ~= your chance of success.)

In theory it seems obviously useful. In practice, it makes people underrate themselves and prematurely give up their ambition. 

One problem is that finding the right comparison group is hard.

For instance, in one commonly-cited statistic that “90% of startups fail,” (https://www.national.biz/2019-small-business-failure-rate-startup-statistics-industry/) “startup” meant all newly-started small businesses including eg groceries. Failure rates vary wildly by industry!

But it’s worse than that. Even if you knew the failure rate specifically of venture-funded tech startups, that leaves out a ton of important info. Consumer startups probably fail at a higher rate than B2B startups. Single-founder startups probably fail more than 2-founder ones.

OK, so what you really need to do is look at the failure rate of 2-founder B2B startups, right?

The problem is that nearly all the relevant information about whether your startup is going to succeed isn’t encoded in your membership in some legible subgroup like that.

Instead, it’s encoded in things like: How likely are you to break up with your cofounder? How good will you be at hiring? How determined are you to never ever give up?

Outside view fans anchor far too strongly on the base rate, and don’t update enough on inside views like these.

“If only 10% of startups succeed, how could I claim a 50% chance? That’d imply I’ve observed evidence with a 5:1 odds ratio! How presumptuous!”

Actually, evidence way stronger than 5:1 is everywhere: markxu.com/strong-evidence

(It’s rarer for startups than for names, but still.)

[...]

According to an answer on Quora, "the real percentage of venture-backed startups that fail—as defined by companies that provide a 1X return or less to investors—has not risen above 60% since 2001" (2017 source). Other Quora answers I saw claimed numbers as high as 98% (on different definitions, like ten-year "failure" (survival?) rate), but didn't cite their sources.

A blurb for the book "The Feeling of Value":

This revolutionary treatise starts from one fundamental premise: that our phenomenal consciousness includes direct experience of value. For too long, ethical theorists have looked for value in external states of affairs or reduced value to a projection of the mind onto these same external states of affairs. The result, unsurprisingly, is widespread antirealism about ethics.

In this book, Sharon Hewitt Rawlette turns our metaethical gaze inward and dares us to consider that value, rather than being something “out there,” is a quality woven into the very fabric of our conscious experience, in a highly objective way. On this view, our experiences of pleasure and pain, joy and sorrow, ecstasy and despair are not signs of value or disvalue. They are instantiations of value and disvalue. When we feel pleasure, we are feeling intrinsic goodness itself. And it is from such feelings, argues Rawlette, that we derive the basic content of our normative concepts—that we understand what it means for something to be intrinsically good or bad.

Rawlette thus defends a version of analytic descriptivism. And argues that this view, unlike previous theories of moral realism, has the resources to explain where our concept of intrinsic value comes from and how we know when it objectively applies, as well as why we sometimes make mistakes in applying it. She defends this view against G. E. Moore’s Open Question Argument as well as shows how these basic facts about intrinsic value can ground facts about instrumental value and value “all things considered.” Ultimately, her view offers us the possibility of a robust metaphysical and epistemological justification for many of our strongest moral convictions.

My reply: Sounds descriptively false? I prefer lots of things that aren't a matter of valenced experience.

You don't need to have "direct experience" of all moral properties in order to be a moral realist, any more than you need direct experience of porcupines in order to be a porcupine realist. You can just acknowledge that moral knowledge is inferred indirectly, the same as our knowledge of most things. Seems like a classic case of philosophers reaching weird conclusions because they're desperate for certainty (rather than embracing Bayesian/probabilistic ways of thinking about stuff).

Likewise, you don't need direct experience or certainty in order to reconcile "is" and "ought". Just accept that "ought" facts look like hypothetical imperatives that we happen to care about a lot, or look like rules-of-a-game that we happen to deeply endorse everyone always playing. No deep riddles are created by the fact that "what is a legal move in chess?" is not reducible to conjunctions of claims about our universe's boundary conditions and laws of physics; we just treat chess-claims like math/logic claims and carry on with our lives. Treating our moral claims (insofar as they're coherent and consistent) in the same way raises no special difficulties.

This also feels to me like an example of the common philosopher-error "notice a super important fact about morality, and rush (in your excitement) to conclude that this must therefore be the important fact about morality, the one big thing morality is About".

Human morality is immensely complicated, because human brains are a complicated, incoherent mishmash of innumerable interacting preferences and experiences. Valenced experience indeed seems to be a super important piece of that puzzle, but we can acknowledge that without pretending that it Solves Everything or Exhausts The Phenomenon.

I suspect moral philosophy would have made a lot more progress by now if philosophers spent more of their time adding to the pool of claims about morality (so we can build a full understanding of what phenomenon we need to explain / account for in the first place), and less time trying to reduce all of those claims to a single simple principle.

In principle, I love theorizing and philosophizing about this stuff. In practice, seems like people have a strong tendency to fall in love with the first Grand Theory of Everything they discover in this domain, causing progress to stagnate (and unrealistic views to proliferate) relative to if we had more modest goals. Less "try to reduce all of morality to virtue cultivation", more "try to marginally improve our understanding of what virtue cultivation consists in".

Ben Weinstein-Raun wrote on social media:

It seems to me that the basic appeal of panpsychism goes like "It seems really weird that you can put together some apparently unfeeling pieces, and then out comes this thing that feels. Maybe those things aren't actually unfeeling? That would sort of explain where the feeling-ness comes from."

But this feels kind of analogous to a being that doesn't have a good theory about houses, but is aware that some things are houses and some things aren't, by their experiences of those things. Such a being might analogously reason that *everything* is a little bit house-y. Panhousism isn't exactly wrong, but it's not actually very enlightening. It doesn't explain how the houseyness of a tree is increased when you rearrange the tree to be a log cabin. In fact it might naively want to deny that the total houseyness is increased.

I think panpsychism is outrageously false, and profoundly misguided as an approach to the hard problem. But I think I can describe it in a way that makes its appeal more obvious:

When I introspect, it seems like there's a lot of complexity to my experiences. But it doesn't seem like a complex fact that that I'm conscious at all -- the distinction between conscious and unconscious seems very basic. It feels like there's 'what it's like to be an algorithm from the inside', and then there's the causal behavior of the algorithm, and that's all there is to it.

And thought experiments show that 'how an algorithm feels from inside' can't be exhaustively reduced to any functional/causal/'external' behavior. (See the knowledge argument, the zombie argument, etc.)

So maybe I should just think of 'algorithms have an inside' as a basic feature of the universe, an extension of 'everything has an inside (so to speak) and the only reason I feel "special" is that I happen to be this one part of the universe'.

Panprotopsychism might make this an easier pill to swallow. We can say that electrons aren't "conscious" in the fashion of a human; but they maybe have an "inside" (in the sense of "how an algorithm feels from inside") in the most rudimentary and abstract way possible, and the more complex "inside" of things like humans is built up out of all these smaller, more bare-bones "insides".

We can imagine that "how an electron feels from inside" is like an empty room. There's an "inside", but there's no content to it, just empty structure. This structure can then produce some amazing things, if you arrange an enormous number of parts in just the right way; but the important thing is to start from the kind of universe that has an "inside" at all, as opposed to the zombie universe.

The problem with this view, as usual, is that we've assumed that the "inside" can't causally affect the "outside"; and yet I just wrote a bunch of paragraphs about the "inside", presumably based on some knowledge I have about that inside.

So either my paragraphs must be straightforwardly false; or I must be confused about what I'm discussing (ascribing attributes to a nonfunctional, nonphysical thing that are really just run-of-the-mill descriptions of physical facts); or my statements must be miraculously true, even though there is no explanation for why I would have any knowledge of the things I'm discussing.

Without some further argument to make sense of the miracle, I think we have to reject the miracle, even if we still feel confused about what's actually going on with phenomenal consciousness.

[Epistemic status: Thinking out loud, just for fun, without having done any scholarship on the topic at all.]

It seems like a lot of horror games/movies are converging on things like 'old people', 'diseased-looking people', 'psychologically ill people', 'women', 'children', 'dolls', etc. as particularly scary.

Why would that be, from an evolutionary perspective? If horror is about fear, and fear is about protecting the fearful from threats, why would weird / uncanny / out-of-evolutionary-distribution threats have a bigger impact than e.g. 'lots of human warriors coming to attack you' or 'a big predator-looking thing stalking you', which are closer to the biggest things you'd want to worry about in our environment of evolutionary adaptedness? Why are shambling, decrepit things more of a horror staple than big bulky things with claws?

(I mean, both are popular, so maybe this isn't a real phenomenon. I at least subjectively feel as though those uncanny things are scarier than super-lions or super-snakes.)

Maybe we should distinguish between two clusters of fear-ish emotions:

1. Terror. This is closer to the fight-or-flight response of 'act quick because you're in imminent danger'. It's a panicky 'go go go go go!!' type of feeling, like when a jumpscare happens or when you're running from a monster in a game.
2. Dread. This is more like feeling freaked out or creeped out, and it can occur alongside terror, or it can occur separately. It seems to be triggered less by 'imminent danger' than by ambiguous warning signs of danger.

So, a first question is why uncanny, mysterious, 'unnatural' phenomena often cause the most dread, even though they thereby become less similar to phenomena that actually posed the largest dangers to us ancestrally. (E.g., big hulking people with giant spears or snakes/dragons or werewolves correlate more with 'things dangerous to our ancestors' than decrepit zombies. Sure, creepiness maybe requires that the threat be 'ambiguous', but then why isn't an ambiguous shadow of a maybe-snake or maybe-hulking-monster creepier than an obviously-frail/sickly monster?)

Plausibly part of the answer is that more mysterious, inexplicable phenomena are harder to control, and dread is the brain's way of saying something like 'this situation looks hard to control in a way that makes me want to avoid situations like this'.

Terror-inspiring things like jumpscares have relatively simple triggers corresponding to a relatively simple response -- usually fleeing. Dread-inspiring things like 'the local wildlife has gotten eerily quiet' have subtler and more context-sensitive triggers corresponding to responses like 'don't necessarily rush into any hasty action, but do pay extra close attention to your environment, and if you can do something to get away from the stimuli that are giving you these unpleasant uneasy feelings, maybe prioritize doing that'.

The wiki glossary for the sequences / Rationality: A-Z ( https://wiki.lesswrong.com/wiki/RAZ_Glossary ) is updated now with the glossary entries from the print edition of vol. 1-2.

New entries from Map and Territory:

anthropics, availability heuristic, Bayes's theorem, Bayesian, Bayesian updating, bit, Blue and Green, calibration, causal decision theory, cognitive bias, conditional probability, confirmation bias, conjunction fallacy, deontology, directed acyclic graph, elan vital, Everett branch, expected value, Fermi paradox, foozality, hindsight bias, inductive bias, instrumental, intentionality, isomorphism, Kolmogorov complexity, likelihood, maximum-entropy probability distribution, probability distribution, statistical bias, two-boxing

New entries from How to Actually Change Your Mind:

affect heuristic, causal graph, correspondence bias, epistemology, existential risk, frequentism, Friendly AI, group selection, halo effect, humility, intelligence explosion, joint probability distribution, just-world fallacy, koan, many-worlds interpretation, modesty, transhuman

A bunch of other entries from the M&T and HACYM glossaries were already on the wiki; most of these have been improved a bit or made more concise.

Jeffrey Ladish asked on Twitter:

Do you think the singularity (technological singularity) is a useful term? I've been seeing it used less among people talking about the future of humanity and I don't understand why. Many people still think an intelligence explosion is likely, even if it's "slow"

I replied:

'Singularity' was vague (https://intelligence.org/2007/09/30/three-major-singularity-schools/) and got too associated with Kurzweilian magical thinking, so MIRI switched to something like:

'rapid capability gain' = progress from pretty-low-impact AI to astronomically high-impact AI is fast in absolute terms

'hard takeoff' = rapid capability gain that's discontinuous

'intelligence explosion' = hard takeoff via recursive self-improvement

Eliezer says:

"'Rapid capability gain' is, I'd say, going from 'capable enough to do moderately neat non-pivotal world-affecting things' to 'capable enough to destroy world' quickly in absolute terms.

"I don't think it's about 'subhuman' because, like, is Alpha Zero subhuman, things go superhuman in bits and pieces until, in some sense, all hell breaks loose"

FOOM is a synonym for intelligence explosion, based on the analogy where an AGI recursively self-improving to superintelligence is like a nuclear pile going critical.

Sometimes people also talk about already-pretty-smart-and-impactful AI "going FOOM", which I take to mean that they're shooting off to even higher capability levels.

Jeffrey replied:

Okay I appreciate these distinctions. I think the difficulty of replacing "singularity" with "intelligence explosion" is that the latter sounds like a process rather than an outcome. I want to refer to the outcome

To which Nate Soares responded:

In my vocab, "singularity" refers to something more like an event (& the term comes from Vinge noting that the dawn of superintelligence obscures our predictive vision). I still use "singularity" for the event, and "post-singularity" for the time regime.

From Facebook:

Mark Norris Lance: [...] There is a long history of differential evaluation of actions taken by grassroots groups and similar actions taken by elites or those in power. This is evident when we discuss violence. If a low-power group places someone under their control it is kidnapping. If they assess their crimes or punish them for it, it is mob justice or vigilanteism. [...]

John Maxwell: Does the low power group in question have a democratic process for appointing judges who then issue arrest warrants?

That's a key issue for me... "Mob rule" is bad because the process mobs use to make their judgements are bad. Doubly so if the mob attacks anyone who points that out.

A common crime that modern mobs accuse people of is defending bad people. But if people can be convicted of defending bad people, that corrupts the entire justice process, because the only way we can figure out if someone really is bad is by hearing what can be said in their defense.

From https://twitter.com/JonHaidt/status/1166318786959609856:

Why are online political discussions perceived to contain elevated levels of hostility compared to offline discussions? In this manuscript, we leverage cross-national representative surveys and online behavioral experiments to [test] the mismatch hypothesis regarding this hostility gap. The mismatch hypothesis entails that novel features of online communication technology induce biased behavior and perceptions such that ordinary people are, e.g., less able to regulate negative emotions in online contexts. We test several versions of the mismatch hypothesis and find little to no evidence in all cases. Instead, online political hostility is committed by individuals who are predisposed to be hostile in all contexts. The perception that online discussions are more hostile seemingly emerges because other people are more likely to witness the actions of these individuals in the large, public network structure of online platforms compared to more private offline settings.

Yeah, I'm an EA: an Estimated-as-Effective-in-Expectation (in Excess of Endeavors with Equivalent Ends I've Evaluated) Agent with an Audaciously Altruistic Agenda.

How would you feel about the creation of a Sequence of Shortform Feeds? (Including this one?) (Not a mod.)

In the context of a conversation with Balaji Srinivasan about my AI views snapshot [LW · GW], I asked Nate Soares what sorts of alignment results would impress him, and he said:

example thing that would be relatively impressive to me: specific, comprehensive understanding of models (with the caveat that that knowledge may lend itself more (and sooner) to capabilities before alignment). demonstrated e.g. by the ability to precisely predict the capabilities and quirks of the next generation (before running it)

i'd also still be impressed by simple theories of aimable cognition (i mostly don't expect that sort of thing to have time to play out any more, but if someone was able to come up with one after staring at LLMs for a while, i would at least be impressed)

fwiw i don't myself really know how to answer the question "technical research is more useful than policy research"; like that question sounds to me like it's generated from a place of "enough of either of these will save you" whereas my model is more like "you need both"

tho i'm more like "to get the requisite technical research, aim for uploads" at this juncture

if this was gonna be blasted outwards, i'd maybe also caveat that, while a bunch of this is a type of interpretability work, i also expect a bunch of interpretability work to strike me as fake, shallow, or far short of the bar i consider impressive/hopeful

(which is not itself supposed to be any kind of sideswipe; i applaud interpretability efforts even while thinking it's moving too slowly etc.)

For being too indistinguishable from GPT-3.