LessWrong 2.0 Reader

By environment, I mean the setting of the scene. Spoken words are sounds in the setting, like the sound of the wind, a gunshot, or an animal’s cry. It just happens that a human voice box is what’s making those particular sounds. McCarthy’s central theme across all the novels of his that I’ve read is the inhumanity of the Mexican-American frontier, and treating human speech as just a sound among other sounds is a key part of how he expresses that theme in his writing style.

Wow, I didn't realize.

Yes, you can ask for a lot more than that :)

This seems like an interesting problem! I've been thinking about it a little bit but wanted to make sure I understood before diving in too deep. Can I see if I understand this by going through the biased coin example?

Suppose I have 2^5 coins and each one is given a unique 5-bit string label covering all binary strings from 00000 to 11111. Call the string on the label $\Lambda$.

The label given to the coin indicates its 'true' bias. The string 00000 indicates that the coin with that label has p(heads)=0. The coin labelled 11111 has p(heads)=1. The ‘true’ p(heads) increases in equal steps going up from 00000 to 00001 to 00010 etc. Suppose I randomly pick a coin from this collection, toss it 200 times and call the number of heads X_1. Then I toss it another 200 times and call the number of heads X_2.

Now, if I tell you what the label on the coin was (which tells us the true bias of the coin), telling you X_1 would not give you any more information to help you guess X_2 (and vice versa). This is the first Natural Latent condition ($\Lambda$ induces independence between X_1 and X_2). Alternatively, if I didn’t tell you the label, you could estimate it from either X_1 or X_2 equally well. This is the other two diagrams.

I think that the full label $\Lambda$ will be an approximate stochastic natural latent. But if we consider only the first bit^[1] of the label (which roughly tells us whether the bias is above or below 50% heads) then this bit will be a deterministic natural latent because with reasonably high certainty, you can guess the first bit of $\Lambda$ from X_1 or X_2. This is because the conditional entropy H(first bit of $\Lambda$|X_1) is low. On the other hand H($\Lambda$ | X_1) will be high. If I get only 23 heads out of 200 tosses, I can be reasonably certain that the first bit of $\Lambda$ is a 0 (ie the coin has a less than 50% of coming up heads) but can't be as certain what the last bit of $\Lambda$ is. Just because $\Lambda$ satisfies the Natural Latent conditions within $ϵ$, this doesn’t imply that $\Lambda$ satisfies $H\left(\Lambda |X_1\right)<ϵ$. We can use X_1 to find a 5-bit estimate of $\Lambda$, but most of the useful information in that estimate is contained in the first bit. The second bit might be somewhat useful, but its less certain than the first. The last bit of the estimate will largely be noise. This means that going from using $\Lambda$ to using ‘first bit of $\Lambda$’ doesn’t decrease the usefulness of the latent very much, since the stuff we are throwing out is largely random. As a result, the ‘first bit of $\Lambda$’ will still satisfy the natural latent conditions almost as well as $\Lambda$. By throwing out the later bits, we threw away the most 'stochastic' bits, while keeping the most 'latenty' bits.

So in this case, we have started from a stochastic natural latent and used it to construct a deterministic natural latent which is almost as good. I haven’t done the calculation, but hopefully we could say something like ‘if $\Lambda$ satisfies the natural latent conditions within $ϵ$ then the first bit of $\Lambda$ satisfies the natural latent conditions within $2ϵ$ (or $3ϵ$ or something else)’. Would an explicit proof of a statement like this for this case be a special case of the general problem?

The problem question could be framed as something like: “Is there some standard process we can do for every stochastic natural latent, in order to obtain a deterministic natural latent which is almost as good (in terms of \epsilon)”. This process will be analogous to the ‘throwing away the less useful/more random bits of \lambda’ which we did in the example above. Does this sound right?

Also, can all stochastic natural latents can be thought of as 'more approximate' deterministic latents? If a latent satisfies the the three natural latents conditions within ${ϵ}_1$, we can always find a (potentially much bigger) ${ϵ}_2$ such that this latent also satisfies the deterministic latent condition, right? This is why you need to specify that the problem is showing that a deterministic natural latent exists with 'almost the same' $ϵ$. Does this sound right?

1. ^{^}

   I'm going to talk about the 'first bit' but an equivalent argument might also hold for the 'first two bits' or something. I haven't actually checked the maths. 

:) the real money was the friends we made along the way.

I dropped out of a math MSc. at a top university in order to spend time learning about AI safety. I haven't made a single dollar and now I'm working as a part time cashier, but that's okay.

What use is money if you end up getting turned into paperclips?

PS: do you want to sign my open letter [LW · GW] asking for more alignment funding?

Exactly. For the purposes of the post I framed it as a single interaction, but my honest response would be 'Cool. Do you have a video?"

While I recommend looking up the real thing or news, in case you are curious I started with a much longer version of the story before I realized it was distracting from the rest of the post:

"So I have an amazing fish. Marcy. She's an archerfish. You know, the kind of fish that can spit at flying insects? She can recognize human faces! You see, I trained her to spit at politicians. 

"We play a game sometimes, where I tap a spot on some glass by the tank. If she hits it, she gets a treat. We'd been doing this for a couple months when I put a TV behind the glass to see if she would react. She didn't really; not at first. I turned it to the news though, and on a lark tapped the glass where that politician was talking. You know, the boring, long-winded one? She spat at 'em and got her treat. 

"So I thought, why not? What if I tap all politicians faces when they come on? It became part of our game. Marcy couldn't predict when I'd tap the glass because I couldn't. She got a game and treats and I got to watch the news. Win-win. 

"But then one day I got bored. I wasn't tapping all the politician faces. It was that day, remember, where the thing happened? News was getting sound bites from everyone and their dog? Pelosi and Trump came up and Marcy nailed 'em both. Dead center. I was already giving her the treat before I realized that I hadn't tapped the glass. 

"So I test her, you know? She spitting at anyone now? Talking heads? Nope. Snoop? Nope. Vance? Dead aim. AOC? A little to the left but Marcy still got a treat. Every politician got hit. Well, not that Canadian guy. But I don't think she's seen that one before. So I figure she can spot faces. I mark the target and she spits 'em when she sees 'em. 

"So what do you say? Come see her, please? I want to know if she'll do it for you, you know? Before I call a fish scientist? Promise it will be a hoot."

I totally agree with this for some/many use cases. I would caution against doing so in the following situations:

1. Novel situations - If a type of situation is new to you, the closest pattern may fail to match well.
2. Unanalyzed situations - If you have always used pattern matching to decide something, it would be valuable to verify the pattern, at least once, to make sure you are not missing or abusing opportunities.
3. Repeated situations - This is a bit trickier, but sometimes we have repeated applications of the same pattern over and over. It would still be useful to analyze these in depth every once in a while, to maintain the calibration of the heuristic. 

In reality it is a balancing act, and it would be best to avoid over-reliance on either approach: over-analysis or pattern heuristics.

In the context of your Twitter comments about biorisk, the relevant question for this post and the appropriateness theory is probably just about whether it would be better to conceptualize these risks as part of a theory that includes a single objective for all humanity ("make the world a better place") versus a theory that doesn't accept that. Our claim is that it's easier to think about 'stability of society' and misuse type harms when using our theory. Nothing you said on Twitter is an argument for why it's important to have anything resembling a single objective.

I do think you're right to point out that the sentence you extracted from the long paper where we speculate about the future doesn't really follow from the theory itself.

Are people intrinsically drawn to having the eye fooled by abstracted drawings?

If you want to apply this to alignment, the next question would be, is there something in the human nature that causes this, or would an AGI likely be drawn to similar effects?

If you gather knowledge about what's intrinsically motivating AGI's as well, that would be valuable for alignment research, because it's about creating motivations for AGI's to do things.

Did these sinister implications go over the heads of audiences, or were they key to the masses’ enjoyment of the songs?

You can reframe that question as, "Is this aspect of the Beatles songs aligned with the desires of the audience?"

Both of your examples about what people or agents in general value. AI alignment is about how to align what humans and AGIs value. Understanding something about the nature of value, seems applicable to AI alignment. 

the main question seems to me to be whether it is plausible that US AI companies would spend more on safety

Other considerations:

• Maybe more sharing between US companies leads to a faster progress of the field overall
  • though maybe it slows it down by reducing investments because investing in algorithmic secrets is less valuable? That's a bit a 4D chess consideration, I don't know how to trust this sort of reasoning.
• Maybe you want information to flow from the most reckless companies to the less reckless ones, but not the other way around, such that you would prefer if the companies you expect to spend the most on safety to not share information. Spending more on info-sec is maybe correlated with spending more on safety in general, therefore you might be disfavoring less reckless actors by asking for transparency.

(I am also unsure how the public will weigh in - I think there is a 2%-20% chance that public pressure is net negative in terms of safety spending because of PR, legal and AI economy questions. I think it's hard to tell in advance.)

I don't think these are super strong considerations and I am sympathetic to the point about safety spend probably increasing if there was more transparency.

The proposal is to use monitoring measures, similar to e.g. constitutional classifiers. Also, don't we reduce misuse risk a bunch by only deploying to 10k external researchers?

My bad, I failed to see that what's annoying with helpful-only model sharing is that you can't check if activity is malicious or not. I agree you can do great monitoring, especially if you can also have a small-ish number of tokens per researcher and have humans audit ~0.1% of transcripts (with AI assistance).