LessWrong 2.0 Reader

But in this Eiffel Tower example, I’m not sure what is correlating with what

The physical object Eiffel Tower is correlated with itself.
 

However, I think the basic ability of an LLM to correctly complete the sentence “the Eiffel Tower is in the city of…” is not very strong evidence of having the relevant kinds of dispositions.

It is highly predictive of the ability of the LLM to book flights to Paris, when I create an LLM-agent out of it and ask it to book a trip to see the Eiffel Tower.
 

I think the question about whether current AI systems have real goals and beliefs does indeed matter

I dont think we disagree here. To clarify, my belief is there are threat models / solutions that are not affected by whether the AI has 'real' beliefs, and there are other threats/solutions where it does matter.

I think CGP Grey perspective puts more weight on Definition 3.

I actually do not understand the distinction between Definition 2 and Definition 3. Don't need to resolve it here. I've editted post to include my uncertainty on this.

It's a beautiful website. I'm sad to see you go. I'm excited to see you write more.

I have found that they mirror you. If you talk to them like a real person, they will act like a real person. Call them (at least Claude) out on their corporate-speak and cheesy stereotypes in the same way you would a person scared to say what they really think.

Neural Network have a bias towards Highly Decomposable Functions. 

tl;dr Neural networks favor functions that can be "decomposed" into a composition of simple pieces in many ways - "highly decomposable functions". 

Degeneracy = bias under uniform prior

Consider a space $W$ of parameters used to implement functions, where each element $w\in W$ specifies a function$f_w:X\to Y$ via some map $\pi$. Here, the set $W$ is our parameter space, and we can think of each $w$ as representing a specific configuration of the neural network that yields a particular function $f_w$. 

The mapping $\pi$ assigns each point $w\in W$ to a function $f_w$. Due to redundancies and symmetries in parameter space, multiple configurations $w$ might yield the same function, forming what we call a fiber, or the "set of degenerates." of $f$ ${\pi }^{-1}\left(f\right)=\{w\in W|\pi (w)=f_w=f\}$

 This fiber is the set of ways in which the same functional behavior can be achieved by different parameterizations. If we uniformly sample from codes, the degeneracy of a function $f$ counts how likely it is to be sampled. 

The Bias Toward Decomposability

Consider a neural network architecture built out of $l$ layers. Mathematically, we can decompose the parameter space $W$ as a product:

$W=W_1\times W_2\times ...\times W_l,$

where each $W_i$ represents parameters for a particular layer. The function implemented by the network, $f_w$, is then a composition:

$f_w=f_{w_1}\circ f_{w_2}\circ ...\circ f_{w_l}$

For a  function $f$ its degeneracy (or the number of ways to parameterize it) is 

$|{\pi }^{-1}\left(f\right)|={\sum }_{(f_1,...,f_l)\in V\left(f\right)}|{\pi }^{-1}\left(f_1\right)|\cdot |{\pi }^{-1}\left(f_2\right)|\cdot ...\cdot |{\pi }^{-1}\left(f_l\right)|$.

Here, $V\left(f\right)$ is the set of all possible decompositions $f=f_1\circ f_2\circ ...\circ f_l$,  of $f$. 

That means that functions that have many such decompositions are more likely to be sampled. 

In summary, the layered design of neural networks introduces an implicit bias toward highly decomposable functions. 

Them: The point of trade is that there are increasing marginal returns to production and diminishing marginal returns to consumption. We specialize in producing different goods, then trade to consume a diverse set of goods that maximizes utility.

Myself: Suppose there were no production possible, just some cosmic endowment of goods that are gradually consumed until everyone dies. Have we gotten rid of the point of trade?

Them: Well if people had different cosmic endowments then they would still trade to get a more balanced set to consume, due to diminishing marginal returns to consumption.

Myself: What if everyone has exactly the same cosmic endowment? And for good measure there are no diminishing returns, the tenth apple produces as much utility as the first.

Them: Well then there's no trade, what's the point? We just consume our cosmic endowment until we run out and die.

Myself: What if I like oranges more than apples, and you like apples more than oranges?

Them: Oh. I can trade one of my oranges for one of your apples, and we will both be better off. Darn it.

Zvi's latest newsletter has a section on this topic! https://thezvi.substack.com/i/151331494/good-advice

I have thought about the issue that you are outlining here and I can't see any possible solutions. Paired with steganography, that strategy indeed undermines that authors' strategy as far as I can see. I haven't been able to come up with any honeytraps either. This seems like a really big problem.

Am I missing something? Maybe one of the authors of the paper can clarify this issue?

I don't have a strong take haha. I'm just expressing my own uncertainty. 

Here's my best reasoning: Under Bayesian reasoning, a sufficiently small posterior probability would be functionally equivalent to impossibility (for downstream purposes anyway). If models reason in a Bayesian way then we wouldn't expect the deductive and abductive experiments discussed above to be that different (assuming the abductive setting gave the model sufficient certainty over the posterior). 

But I guess this could still be a good indicator of whether models do reason in a Bayesian way. So maybe still worth doing? Haven't thought about it much more than that, so take this w/ a pinch of salt. 

Ah I see ur point. Yh I think that’s a natural next step. Why do you think it not very interesting to investigate? Being able to make very accurate inferences given the evidence at hand seems important for capabilities, including alignment relevant ones?

Historically, I've found that LW comments have been a source of anxious and/or irritated rumination. That's why I mostly haven't commented this year. I'll write more about this in another post.

If I write these days, I generally don't read replies. (Again, excepting certain posts; and I'm always reachable via email and enjoy thoughtful discussions :) )