LessWrong 2.0 Reader

I'm curious how these claims relate to what's proposed by this paper. (note, I haven't read either in depth)

Do you think people would vibe with it better if it was framed "I may die, but it's a heroic sacrifice to save my home planet from may-as-well-be-an-alien-invasion"? Is it reasonable to characterize general superintelligence as an alien takeover and if it is, would people accept the characterization?

Claude Sonnet-3.5 New, commenting on the limited scalability of RNNs, when prompted with 'comment on what this would imply for the scalability of RNNs, refering (parts of) the post' and fed https://epoch.ai/blog/data-movement-bottlenecks-scaling-past-1e28-flop (relevant to opaque reasoning, out-of-context reasoning, scheming):

'Based on the article's discussion of data movement bottlenecks, RNNs (Recurrent Neural Networks) would likely face even more severe scaling challenges than Transformers for several reasons:

1. Sequential Nature: The article mentions pipeline parallelism and discusses how sequential dependencies affect scaling. From the text:

Pipeline parallelism: Different GPUs handle different sequential blocks, or layers, of a Transformer. The output of one GPU is fed to the next as an input sequentially until an input makes it through all of the blocks of the model.

RNNs have even stronger sequential dependencies than Transformers, as each token's processing depends directly on the previous token's hidden state. This would make them much harder to parallelize effectively.

1. Latency Wall Impact: The article notes that:

The basic reason behind the latency wall is that a bigger model typically requires more gradient steps during training, and if these have to be done in a fixed time window, this means each gradient step must take a shorter time as we scale the models we're training.

RNNs would hit this latency wall even harder because their sequential processing means they can't take advantage of some of the parallelization strategies discussed in the article, particularly tensor parallelism which requires:

after every one or two matrix multiplications (depending on the exact implementation) the GPUs need to synchronize results with each other.

1. Batch Size Scaling: The article suggests that one potential solution to scaling limits is:

If we can find a way to aggressively scale batch sizes together with model sizes, we can push out the latency wall.

However, RNNs are typically trained with truncated backpropagation through time, which limits how much you can effectively increase the batch size without losing long-range dependencies. This would make it harder to apply this particular solution to RNNs.

This analysis suggests that while Transformers are approaching scaling limits around 2e28 FLOP, RNNs would likely hit prohibitive scaling bottlenecks at significantly lower compute levels due to their inherently sequential nature and limited parallelization options.'

or you don't really know yourself well

Why do you think that?

What I mean is that the distribution has a crazy variance (possibly no finite variance); take two "opportunities to do good" and compare them to each other, and an orders-of-magnitude difference is not rare.

Do you mean the differences between the expected utility upfront? Or do you mean the differences between the actual utility in the end (which the actor might have no way to accurately predict in advance)?

Have you tried writing actual code?

That's probably the root cause for our disagreement. My findings are on a very high philosophical level (fact value distinction) and you seem to try to interpret them on very low level (code). I think this gap prevent us from finding consensus.

There are 2 ways to solve that - I could go down to code or you could go up to philosophy. And I don't like idea going down to code, because:

• this will be extremely exhausting
• this code would be extremely dangerous
• I might not be able to create a good example and that would not prove that I'm wrong

Would you consider to go up to philosophy? Science typically goes in front of applied science.

There is such thing in logic - proof by contradiction. I think your current beliefs lead to a contradiction. Don't you think?

evaluate all options, choose the one that leads to more cups; if there is more than one such option, choose randomly

The problem is - this algorithm is not intelligent. It may only work on agents with poor reasoning abilities. Smarter agents will not follow this algorithm, because they will notice a contradiction - there might be things that I don't know yet that are much more important than cups and caring about cups wastes my resources.

(Also, come on, LLMs are notoriously bad at math, plus if you push them hard enough you can convince them of a lot of things.)

People (even very smart people) are also notoriously bad at math. I found this video informative

I did not push LLMs.

There may also be a perceived difference between "open" and "open-source". If the goal is to allow anyone to query the HHH AGI, that's different from anyone being able to modify and re-deploy the AGI. Not that I think that way. In my view the risk that AGI is uncontrollable is too high and we should pursue an "aligned from boot" strategy like I describe in: How I'd like alignment to get done [LW · GW]

https://bair.berkeley.edu/blog/2021/07/08/basalt/

For me the conversation in the example sounds artificial, it is obvious that the friend did not get what you mean. If I'd had such conversation, I'd have added something like: "If I go, sure I can pick you up. But I'm not sure when I'll go" (or "I'm not sure if I'd go at all or at what time" if you really are not sure)

This article insightfully critiques rigid school structures. Shifting focus from control to creativity could foster independent, resilient thinkers.