LessWrong 2.0 Reader

Some details mildly off, but I think you've got the big picture basically right.

Alternatively, if I didn’t tell you the label, you could estimate it from either X_1 or X_2 equally well. This is the other two diagrams.

Minor clarification here: the other two diagrams say not only that I can estimate the label equally well from either $X_1$ or $X_2$, but that I can estimate the label (approximately) equally well from $X_1$, $X_2$, or the pair $(X_1,X_2)$.

I think that the full label $\Lambda$ will be an approximate stochastic natural latent.

I'd have to run the numbers to check that 200 flips is enough to give a high-confidence estimate of $\Lambda$ (in which case 400 flips from the pair of variables will also put high confidence on the same value with high probability), but I think yes.

But if we consider only the first bit^[1] of the label (which roughly tells us whether the bias is above or below 50% heads) then this bit will be a deterministic natural latent because with reasonably high certainty, you can guess the first bit of $\Lambda$ from $X_1$ or $X_2$.

Not quite; I added some emphasis. The first bit will (approximately) satisfy the two redundancy conditions, i.e. $X_1\to X_2\to \Lambda$ and $X_2\to X_1\to \Lambda$, and indeed will be an approximately deterministic function of $X$. But it won't (approximately) satisfy the mediation condition $X_1\leftarrow \Lambda \to X_2$; the two sets of flips will not be (approximately) independent given only the first bit. (At least not to nearly as good an approximation as the original label.)

That said, the rest of your qualitative reasoning is correct. As we throw out more low-order bits, the mediation condition becomes less well approximated, the redundancy conditions become better approximated, and the entropy of the coarse-grained latent given $X$ falls.

So to build a proof along these lines, one would need to show that a bit-cutoff can be chosen such that bit_cutoff($\Lambda$) still mediates (to an approximation roughly $ϵ$-ish), while making the entropy of bit_cutoff($\Lambda$) low given $X$.

I do think this is a good angle of attack on the problem, and it's one of the main angles I'd try.

If a latent satisfies the the three natural latents conditions within ${ϵ}_1$, we can always find a (potentially much bigger) ${ϵ}_2$ such that this latent also satisfies the deterministic latent condition, right? This is why you need to specify that the problem is showing that a deterministic natural latent exists with 'almost the same' $ϵ$. Does this sound right?

Yes. Indeed, if we allow large enough $ϵ$ (possibly scaling with system size/entropy) then there's always a deterministic natural latent regardless; the whole thing becomes trivial.

I agree that one part of our conclusion is related to corrigibility. We didn't use that term here but we have done so elsewhere, in older work from before we had the new theory. We could probably still use it. It might be a distraction to this audience though since corrigibility looks rather different in a multi-agent context, and it's not the focus here.

I've been working on the reverse direction: chopping up $P\left[\Lambda \right]$ by clustering the points (treating each distribution as a point in distribution space) given by $P[\Lambda |X=x]$, optimizing for a deterministic-in-$X$ latent $\Delta =\Delta \left(X\right)$ which minimizes $D_{KL}\left(P\right[\Lambda |X\left]||P\right[\Lambda |\Delta \left(X\right)\left]\right)$.

This definitely separates $X_1$ and $X_2$ to some small error, since we can just use $\Delta$ to build a distribution over $\Lambda$ which should approximately separate $X_1$ and $X_2$.

To show that it's deterministic in $X_1$ (and by symmetry $X_2$) to some small error, I was hoping to use the fact that---given $X_1$---$X_2$ has very little information about $\Lambda$, so it's unlikely that $P\left[\Lambda |X_1\right]$ is in a different cluster to $P[\Lambda |X_1,X_2]$. This means that $P\left[\Delta |X_1\right]$ would just put most of the weight on the cluster containing $P\left[\Lambda |X_1\right]$.

A constructive approach for $\Delta$ would be marginally more useful in the long-run, but it's also probably easier to prove things about the optimal $\Delta$. It's also probably easier to prove things about $\Delta$ for a given number of clusters $|\Delta |$, but then you also have to prove things about what the optimal value of $|\Delta |$ is.

I agree wholeheartedly with the sentiment. I also agree with the underlying assumptions made in the Compendium^[1], that it would really require a Manhattan project level of effort to understand: 

1. What intelligence actually is and how it works, as we don't yet have a robust working theory of intelligence.
2. What alignment actually looks like, and how we can even begin to formulate a thesis of how to keep a superintelligent system aligned as it evolves and recursively self improves. I liken this a bit to the hard problem of consciousness. It's the hard problem of alignment, which I parse into two discreet components:
   1. We don't understand what drives humans completely. We expect a degree of 'playing nice' from AI systems, but we don't have a robust and provable theory of why humans are social, self-sacrifice, sometimes think of the common good, sometimes don't, are curious, and which factors of our experience (or our biology) are responsible for those drivers. Without that, attempting to simulate them in an AI system seems like a dead end. Surface level alignment is trivial (it's polite and friendly), real alignment could potentially be intractable, as we don't have a working theory of what aligns humans to begin with.
   2. We require more than just basic 'behave like a human' level of alignment from an AI system. Humans do an incredible amount of harm, both to each other (war, exploitation, famine), and to the natural World (habitat destruction, pollution, etc.) in the pursuit of our goals. We need a model for behavior that transcends that human behavior. Which leads to the question of, how is that goal even to be formulated? How is a set of behaviors, goals and values instilled in an AI system, from a species that does not routinely possess those goals?
3. AI systems in their current form, in order to increase in capabilities, exacerbate many of the issues that are caused by humans. They are extremely power intensive. Power that at the moment is only realistically servable by fossil fuels. That power has to be 'stable' and 'dispatchable'. This is not the power profile of most renewables, which are inherently cyclical, and irregular in their generation. While we may not be concerned about inference stopping due to a BESS system running out of charge, a superintelligent system relying on that power for its existence may think differently.
4. AI systems also require more rare earth elements, commodities, etc. that are difficult and environmentally challenging to extract. In water scarse times, they require a lot of water for cooling, which is impacting local communities. In order for a superintelligent system to grow rapidly in the short term, it must continue to consume these resources in huge quantities, putting it in direct, and inherent conflict with any alignment goal around the preservation of the natural world, and the ecosystem that humans are reliant upon. Given the pursuit of resources and living space is the fundamental driver of most human conflict - it would appear that we could be setting ourselves up for a resource conflict with AI. How to resolve this, is something that I don't think anyone has a clear answer to, and I don't see being discussed at all often enough. (Perhaps because I work in the data center infrastructure space, I find myself running into this almost daily, so its front of mind for me). 

All of which is to say, that I believe these problems are resolvable, but only if, to your point, a significant amount of expenditure, and the greatest minds in this generation are set to the task of resolving them ahead of the deployment of a superintelligent system. We face a Manhattan Project level of risk, but we are not acting as if we are facing that systematically. 

1. ^{^}

   The Compendium

1. Not sure what you mean by this comment.
2. Doesn't this hindsight-based definition of CEV that you offer here preclude using it as an objective to coordinate around? You can't coordinate around something that you'll only know in hindsight. And however you try to estimate it, if you try to get people to coordinate around your estimated CEV than some groups will feel like it doesn't represent their view of how to make the world a better place, or it doesn't prioritize X over Y in the way they would, etc.

Oh, you're right.

By environment, I mean the setting of the scene. Spoken words are sounds in the setting, like the sound of the wind, a gunshot, or an animal’s cry. It just happens that a human voice box is what’s making those particular sounds. McCarthy’s central theme across all the novels of his that I’ve read is the inhumanity of the Mexican-American frontier, and treating human speech as just a sound among other sounds is a key part of how he expresses that theme in his writing style.

Wow, I didn't realize.

Yes, you can ask for a lot more than that :)

This seems like an interesting problem! I've been thinking about it a little bit but wanted to make sure I understood before diving in too deep. Can I see if I understand this by going through the biased coin example?

Suppose I have 2^5 coins and each one is given a unique 5-bit string label covering all binary strings from 00000 to 11111. Call the string on the label $\Lambda$.

The label given to the coin indicates its 'true' bias. The string 00000 indicates that the coin with that label has p(heads)=0. The coin labelled 11111 has p(heads)=1. The ‘true’ p(heads) increases in equal steps going up from 00000 to 00001 to 00010 etc. Suppose I randomly pick a coin from this collection, toss it 200 times and call the number of heads X_1. Then I toss it another 200 times and call the number of heads X_2.

Now, if I tell you what the label on the coin was (which tells us the true bias of the coin), telling you X_1 would not give you any more information to help you guess X_2 (and vice versa). This is the first Natural Latent condition ($\Lambda$ induces independence between X_1 and X_2). Alternatively, if I didn’t tell you the label, you could estimate it from either X_1 or X_2 equally well. This is the other two diagrams.

I think that the full label $\Lambda$ will be an approximate stochastic natural latent. But if we consider only the first bit^[1] of the label (which roughly tells us whether the bias is above or below 50% heads) then this bit will be a deterministic natural latent because with reasonably high certainty, you can guess the first bit of $\Lambda$ from X_1 or X_2. This is because the conditional entropy H(first bit of $\Lambda$|X_1) is low. On the other hand H($\Lambda$ | X_1) will be high. If I get only 23 heads out of 200 tosses, I can be reasonably certain that the first bit of $\Lambda$ is a 0 (ie the coin has a less than 50% of coming up heads) but can't be as certain what the last bit of $\Lambda$ is. Just because $\Lambda$ satisfies the Natural Latent conditions within $ϵ$, this doesn’t imply that $\Lambda$ satisfies $H\left(\Lambda |X_1\right)<ϵ$. We can use X_1 to find a 5-bit estimate of $\Lambda$, but most of the useful information in that estimate is contained in the first bit. The second bit might be somewhat useful, but its less certain than the first. The last bit of the estimate will largely be noise. This means that going from using $\Lambda$ to using ‘first bit of $\Lambda$’ doesn’t decrease the usefulness of the latent very much, since the stuff we are throwing out is largely random. As a result, the ‘first bit of $\Lambda$’ will still satisfy the natural latent conditions almost as well as $\Lambda$. By throwing out the later bits, we threw away the most 'stochastic' bits, while keeping the most 'latenty' bits.

So in this case, we have started from a stochastic natural latent and used it to construct a deterministic natural latent which is almost as good. I haven’t done the calculation, but hopefully we could say something like ‘if $\Lambda$ satisfies the natural latent conditions within $ϵ$ then the first bit of $\Lambda$ satisfies the natural latent conditions within $2ϵ$ (or $3ϵ$ or something else)’. Would an explicit proof of a statement like this for this case be a special case of the general problem?

The problem question could be framed as something like: “Is there some standard process we can do for every stochastic natural latent, in order to obtain a deterministic natural latent which is almost as good (in terms of \epsilon)”. This process will be analogous to the ‘throwing away the less useful/more random bits of \lambda’ which we did in the example above. Does this sound right?

Also, can all stochastic natural latents can be thought of as 'more approximate' deterministic latents? If a latent satisfies the the three natural latents conditions within ${ϵ}_1$, we can always find a (potentially much bigger) ${ϵ}_2$ such that this latent also satisfies the deterministic latent condition, right? This is why you need to specify that the problem is showing that a deterministic natural latent exists with 'almost the same' $ϵ$. Does this sound right?

1. ^{^}

   I'm going to talk about the 'first bit' but an equivalent argument might also hold for the 'first two bits' or something. I haven't actually checked the maths.