LessWrong 2.0 Reader

What exactly is worrying about AI developing a comprehensive picture of your life? (I can think of at least a couple problems, e.g. privacy, but I'm curious how you think about it)

I think when explaining it to non-technical people, just saying something like “it’s a big next word predictor” is close enough to the truth to work.

One of the reasons why it's plausible that today's or tomorrow's LLMs can result in brief simulations of consciousness or even qualia is that it happens with dreams in humans. Dreams are likely some sort of processing of information/compression/garbage collection, yet they still result in (badly) simulated experiences as a clear side-effect of trying to work with human experience data.

To be clear, I think it’s very unlikely they are conscious etc., this is a comment on a reflexive process going on in my head

I find that the new personalities of 4o trigger my “person” detectors too much, and I feel uncomfortable extracting work from them. 

I don't know about "by a paperclip maximizer", but one thing that stands out to me:

If we're in a simulation, we could be in a simulation where the simulator did 1e100 rollouts from the big bang forward, and then collected statistics from all those runs.

But we could also be in a simulation where the simulator is doing importance sampling - that is, doing fewer rollouts from states that tend to have very similar trajectories given mild perturbations, and doing more rollouts from states that tend to have very different trajectories given mild perturbations.

If that's the case, we should find ourselves living in a world where events seem to be driven by coincidences and particularly by things which are downstream of chaotic dynamics and which had around a 50/50 chance of happening vs not. We should find more such coincidences for important things than for unimportant things.

Hey, maybe it's not our fault we live in the clown world. Maybe the clown world is a statistical inevitability.

Yet the universe runs on strikingly simple math (relativity, quantum mechanics); such elegance is exactly what an efficient simulation would use. Physics is unreasonably effective, reducing the computational cost of the simulation. This cuts against the last point.

This does not seem so consistent, and if the primary piece of evidence for me against such simulation arguments. I would imagine simulations targeting, eg, a particular purpose would have their physics tailored to that purpose much more than ours seems to (for any purpose, given the vast computational complexity of our physics, and the vast number of objects such a physics engine needs to keep track of). For example, I'd expect most simulations physics to look more like Greg Egan's Crystal Nights (incidentally this story is what first convinced me the simulation hypothesis was false).

One may argue its all there just to convince us we're not in a simulation. Perhaps, but two points:

1. Given the discourse on the simulation hypothesis, most seem to take our physics as evidence in favor of it, as you do here. So I don't think most think clearly enough about this for our civilizational decisions to be so dependent on this.

2. The simulators will have trade-offs and resource constraints too. Perhaps they simulate few highly detailed simulations, and many highly simplified simulations. If this is exponential, in the sense that as the detail decreases the number of simulations exponentially increases, we should expect to be in the least detailed world consistent with the existence of sentiences and for which its not blatantly obvious we're in a simulation.

Of course this argument would break given sufficiently different physics from ours, enabling perhaps our world to be simulated in as much depth as it is very cheaply. But then that seems intuitively at least very unlikely & complex a hypothesis.

Thanks! Yep, makes sense - that's one of the things we'll be working on and hope to share some results soon!

I mean this is a summary of a talk I didn't see so I want to reserve some judgement. But at the same time I just can't imagine being in a situation where I don't want to use the word signal because the other person might think I'm talking about cell signal, so instead I bust out "kodo"

In general, I think we should be pretty wary of taking basic ideas and dressing them up in fancy words. It serves no purpose other than in-group signaling.

I think there is a difference between finetuning and prompting in that in the prompting case, the LLM is aware that it's taking part in a role playing scenario. With finetuning on synthetic documents, it is possible to make the LLM more deeply believe something. Maybe one could make the finetuning more sample efficient by instead distilling a prompted model. Another option could be using steering vectors, though I'm not sure that would work better than prompting.