LessWrong 2.0 Reader

One is introspecting on your current mental state ("I feel a headache starting")

That's mostly what I had in mind as well. It still implies the ability to access a hierarchical model of your current state.

You're not just able to access low-level facts like "I am currently outputting the string 'disliked'", you also have access to high-level facts like "I disliked the third scene because it was violent", "I found the plot arcs boring", "I hated this movie", from which the low-level behaviors are generated.

Or using your example, "I feel a headache starting" is itself a high-level claim. The low-level claim is "I am experiencing a negative-valence sensation from the sensory modality A of magnitude X", and the concept of a "headache" is a natural abstraction over a dataset of such low-level sensory experiences.

It sounds like the core idea is a variant of the Intelligence Manhattan Project idea, but with a focus on long term international stability & a ban on competitors.

Perhaps the industry would be more likely to adopt this plan if GUARD could seek revenue the way corporations currently do: by selling stock & API subscriptions. This would also increase productivity for GUARD & shorten the dangerous arms race interval.

I have a pedantic and a non-pedantic answer to this. Pedantic: you say X is "usually considered good" if it increases welfare. Perhaps you mean to imply that if X is usually considered good then it is good. In this case, I refer you to the rest of the paragraph you quote.

Non-pedantic: yes, it's true that once you accept some fundamental assumptions about goodness and badness you can go about theorising and looking for evidence. I'm suggesting that motivated reasoning is the mechanism that makes those fundamental assumptions believable.

I added a paragraph mentioning this, because I think your reaction is probably common.

The information/assurance split feels quite familiar to me as an engineering manager.

My work life revolves around projects, especially big projects that takes months to complete. Other parts of the business depend on when these projects will be done. In some cases, the entire company's growth plans may hinge on my team completing a project by a certain time. And so everyone wants as much assurance as possible about when projects will complete.

This makes it really hard to share information, because people are so hungry for assurance they will interpret almost any sharing of information as assurance. A typical conversation I used to have when I was naive to this fact:

Sales manager: Hey, Gordon, when do you think that project will be done?

Me: Oh, if things go according to plan, probably next month.

Sales manager: Cool, thanks for the update!

If the project ships next month, no problem. But as often happens in software engineering, if the project gets delayed, now the sales manager is upset:

Them: Hey, you said it would be ready next month. What gives?

Me: I said if things went according to plan, but there were surprises, so it took us longer than we initially though it would.

Them: Dammit. I sold a customer on the assumption that the project was shipping this month! What am I supposed to tell them now?

Me: I don't know, why did you do that? I was giving you an internal estimate, not a promise of delivery.

Them: You said this month. I'm tired of Engineering always having some excuse about why stuff is delayed.

What did I do wrong? I failed to understand that Sales, and most other functions in a software business, are so dependent and hungry for information from Engineering, that they saw the assurance they wanted to see rather than the information I was giving.

I've (mostly) learned my lesson. I have to carefully control how much I say to anyone not directly involved in the project, lest they get the wrong idea.

Someone: Hey, Gordon, when do you think that project will be done?

Me: We're working on it. We set a goal of having it complete by end of next quarter.

Do I actually expect it to take all the way to next quarter? No. Most likely it'll be done next month. But if anything unexpected happens, now I've given a promise I can keep.

This isn't exactly just "underpromise, overdeliver". That's part of it, but it's also about noticing when you're accidentally making a promise, even when you think you're not, even if you say really explicitly that you're not making a promise, someone will interpret as a promise and now you'll have to deal with that.

Yes, certain places like preschools might benefit even from an isolated install.

But that is kind of exceptional.

The world isn't an efficient market, especially because people are kind of set in their ways and like to stick to the defaults unless there is strong social pressure to change.

We believe many things because we are somewhat rational; we consider hypotheses, compare them with observed evidence, and emphasise those that are more compatible. Goodness and badness defy this practice; normal reasoning does not produce hypotheses of the form "if X is morally good then I will observe Y".

I disagree. If X is an action, is is usually considered good if it increases welfare, and bad if it decreases welfare. And we can find evidence for or against something being conducive to welfare. So we can find evidence for or against something being good.

If you installed it in a preschool and it successfully killed all the pathogens there wouldn't be essentially no effect.

I think with sunglasses there’s a veneer of plausible deniability. They in fact have a utilitarian purpose outside of just creating information asymmetry. If you’re wearing a mask though, there’s no deniability. You just don’t want people to know where you’re looking.

It’s a cute idea, but AI is a terrible fiction writer.

That's a good counterexample ! Masks are dangerous and mysterious, but not cool in the way sunglasses are in agree