LessWrong 2.0 Reader

There are much better ways of betting on your beliefs about the valuations of AI firms over the next year than wagering with people you met on Less Wrong. See this post by Ege [LW · GW] for more. 

I'll attempt to keep it short

I started reasoning with this observation that applies to all problem-solving algorithms: The more knowledge^[1] you have, the more deterministic the solution is. In ML, the missing knowledge encoded in the weights is the premise of the entire science. So the role of it is to fill in the gaps of our understanding

How much randomness? The key point is: We want to use as much useful unique knowledge to solve the problem. So we use as much knowledge as we already have(architecture, transfer learning, etc.), and if randomness can give us understanding we don't have, we use it as a substitute

I think this generalizes to evolution. Mind you, the world started with no prior knowledge^[2]. And so the properties^[3] of the evolutionary algorithm make total sense when understood through this lens

There is a vague connection to BPP vs P: only in the realm of when we don't have complete knowledge of the problem, so my idea doesn't really help in this regard

1. ^{^}

   On all abstraction levels: knowledge about the problem, the chosen paradigm, architecture/technique, optimization techniques, parameters

2. ^{^}

   Some could say the laws of physics and matter is a form of knowledge that allowed everything else to unfold

3. ^{^}

   Because the search isn't defined and constrained by prior knowledge, the search space is broader and the algorithm is slower

Yeah, good point. I changed it to "in America in 1850 it would have been taboo to say that there is nothing wrong with interracial relationships."

I'm kind of baffled that people are so willing to say that LLMs understand X, for various X. LLMs do not behave with respect to X like a person who understands X, for many X.

Do you have two or three representative examples?

The post setup skips the "AIs are loyal to you" bit, but it does seem like this line of thought broadly aligns with the post.

I do think this does not require ASI, but I would agree that including it certainly doesn't help.

In particular, even if the LLM were being continually trained (in a way that's similar to how LLMs are already trained, with similar architecture), it still wouldn't do the thing humans do with quickly picking up new analogies, quickly creating new concepts, and generally reforging concepts.

Is this true? How do you know? (I assume there's some facts here about in-context learning that I just happen to not know.)

It seems like eg I can teach an LLM a new game in one session, and it will operate within the rules of that game.

My intuition says reward hacking seems harder to solve than this (even in EEA), but I'm pretty unsure. One example is, under your theory, what prevents reward hacking through forming a group and then just directly maxing out on mutually liking/admiring each other?

When applying these ideas to AI, how do you plan to deal with the potential problem of distributional shifts happening faster than we can edit the reward function?

Remmelt, if you wish, I'm happy to operationalize a bet. I think you're wrong.

wildly parallel thinking and prototyping. i'd hop on a call.

Remember that we have no a priori reason to suspect that there are jumps in the future; humans perform sequential reasoning differently, so comparisons to the brain are just not informative.

In what way do we do it differently than the reasoning models?