LessWrong 2.0 Reader

Scaffolded LLMs are pretty good at not just writing code, but also at refactoring it. So that means that all the tech debt in the world will disappear soon, right?

I predict "no" because

• As writing code gets cheaper, the relative cost of making sure that a refactor didn't break anything important goes up
• The number of parallel threads of software development will also go up, with multiple high-value projects making mutually-incompatible assumptions (and interoperability between these projects accomplished by just piling on more cose).

As such, I predict an explosion of software complexity and jank in the near future.

If preliminary results on the poll hold, then that would be pretty in line with my hypothesis of most people preferring creating simulations with no suffering over a world like ours. However, it is pretty important to note that this might not be representative of human values in general, because looking at your Twitter account, your audience comes mostly from a very specific circles of people (those interested in futurism and AI).

Would someone else reporting to have experienced intense suffering decrease your credence in being in a simulation?

No. Memory about intense sufferings are not intense.

I was mostly trying to approach the problem from a slightly different angle. I wasn't meant to suggest that memories about intense suffering are themselves intense.

As far as I understand it, your hypothesis was that Friendly AI temporarily turns people into p-zombies during moments of intense suffering. So, it seems that someone experiencing intense suffering while conscious (p-zombies aren't conscious) would count as evidence against it.

Reports of conscious intense suffering are abundant. Pain from endometriosis (a condition that affects 10% of women in the world) has been so brutal that it made completely unrelated women tell the internet that their pain was so bad they wanted to die (here and here).

If moments of intense suffering were replaced by p-zombies, then these women would've just suddenly lost consciousness and wouldn't have told the internet about their experience.

From their perspective, it would've look like this: as the condition progresses, the pain gets worse, and at some point, they lose consciousness, only to regain it when everything is already over. They wouldn't have experienced the intense pain that they reported to have experienced. Ditto for all PoWs who have experienced torture.

Yes, only moments. The badness of not-intense sufferings is overestimated, in my personal view, but this may depend on a person.

That's a totally valid view as far as axiological views go, but for us to be in your proposed simulation, the Friendly AI must also share it. After all, we are imagining a situation where it goes on to perform a complicated scheme that depends on a lot of controversial assumptions. To me, that suggests that AI has so many resources that it wouldn't feel bad about one of the assumptions turning out to be false and losing all the invested resources. If the AI has that many resources, I think it isn't unreasonable to ask why it didn't prevent suffering that is not intense (at least in a way I think you are using the word) but is still very bad, like breaking an arm or having a hard dental procedure without anesthesia.

This Friendly AI would have a very peculiar value system. It is utilitarian, but it has a very specific view of suffering, where suffering basically doesn't count for much below a certain threshold. It is seemingly rational (Friendly AI that managed to get its hand on so many resources should possess at least some level of rationality), but chooses to go for a highly risky and relatively costly plan of Ressurection Simulation over just creating simulations that are maximally efficient at converting resources into value.

There is another somewhat related issue. Imagine a population of Friendly AIs that consist of two different versions of Friendly AI, both of which really like the idea of simulations.

Type A: AIs that would opt for Ressurection Simulation.

Type B: AIs that would opt for simulations that are maximally efficient at converting resources into value.

Given the unnecessary complexity of our world (all of the empty space, quantum mechanics, etc), it seems fair to say that Type B AIs would be able to simulate more humans, because they would have more resources left for this task (Type A AIs are spending some amount of their resources on the aforementioned complexity). Given plausible anthropics and assuming that the number of Type A AIs is equal to the number of Type B AIs in our population, we would expect ourselves to be in a simulation by Type B AI (but we are, unfortunately, not).

For us to be in a Ressurection Simulation (just focusing on these two types of value systems a future Friendly AI might have), there would have to be more Type A AIs than Type B AIs. And I think this fact is going to be very hard to prove. And this isn't me being nitpicky; Type B AI is genuinely much closer to my personal value system than Type A AI.

More generally speaking, what you presenting as global showstoppers, are technical problems that can be solved.

I don't think the simulations that you described are technically impossible. I am not even necessarily against simulations in general. I just think that, given observable evidence, we are not that likely to be in either of the simulations that you have described.

We de-empahsized QM in the post

You did a bit more than de-emphasize it in the title!

@quiet_NaN

If the authors claim that adding randomness in the territory in classical mechanics requires making it more complex, they should also notice that for quantum mechanics, removing the probability from the territory for QM (like Bohmian mechanics) tends to make the the theories more complex.

People here tend to prefer many worlds .. it isn't clear that it's more complex ... but it isn't clear that it is a actually simpler than collapse theories, as advertised here.

I'd be happy to give you good odds on, conditional on this policy being enacted, it not expanding to comprise more than 0.1% of total US taxation.

I don't trust my measurements as much in the stubble case, because of the risk of particles leaking into the bag through its exit. So presenting the other cases as relative to stubble risks compounding error.

If the relevant counterfactual is not masking, then I think I'm giving these reductions the right way around?

This is a very nice meta-level discussion of why consciousness discourse gets so bad, and I do genuinely appreciate trying to get cruxes and draw out the generators of a disagreement, which is useful in difficult situations.

One factor that is not really discussed, but amplifies the problem of discourse around consciousness is that people use the word consciousness to denote a scientific and a moral thing, and people often want to know the answer to whether something is conscious because they want to use it to determine whether uploading is good, or whether to care about someone, and way too much discourse does not decouple these 2 questions.

I actually slightly voted against the linked post below in the review, due to methodological problems, but I have a high prior that something like this is a huge contributor to consciousness discourse sucking, and this is an area where the science questions need to be decoupled from value questions:

https://www.lesswrong.com/posts/KpD2fJa6zo8o2MBxg/consciousness-as-a-conflationary-alliance-term-for [LW · GW]

+9 for drawing out a generator on a very confusing topic, and should be in the LW canon for how to deal with difficult disagreements as a worked example.

I'm not going to review the object level on what consciousness actually is, because I already did that in a different review linked below, but the sneak peek is that I'm in camp 1, though you could also call me a camp 2 person, but notably reductionist/computationalist rather than positing novel metaphysics:

https://www.lesswrong.com/posts/FQhtpHFiPacG3KrvD/seth-explains-consciousness#7ncCBPLcCwpRYdXuG [LW(p) · GW(p)]

Hm, I disagree with this, if only because I think that assuming we are eliminativist about identity for present day humans, then uploading vacuously preserves identity, because there's nothing there to preserve, for the same reason as 0-0=0

And I also think that for uploading purposes, I don't believe that uploading does anything special to your consciousness/identity, in that uploading doesn't suddenly cause you to have or lose your identity, due to my answer above.

Given this, why then do I merely doubt the presence of phenomenal consciousness instead of being certain of its absence?

Because a reseacher friend pointed out to me that one could interpret RL training as supplying a sort of 'valence'.

I think the most direct way to clear up this confusion is through experiments.

This paper shows how you can explicitly give a transformer model access to its internal state: https://scholar.google.com/citations?view_op=view_citation&hl=en&user=LKv32bgAAAAJ&sortby=pubdate&citation_for_view=LKv32bgAAAAJ:dQ2og3OwTAUC

This paper shows how you can make the model's loss function dependent on both predicting the future and predicting itself: https://arxiv.org/abs/2407.10188

So I think these give us a place to start. Do we notice qualitative shifts in behavior following a fine-tuning regime which combines these techniques?

I have ideas for further experiments at well.

I have, but my writings are pretty disorganized at the moment and probably hard for people to interpret without some sort of dialogue with me, which is probably why I invited Nicholas/Heather to message me (I no longer remember my exact thought process from a year ago when I wrote the grandparent comment). But regardless, here are some links that you can check out of learning-related content I have written (feel free to message me or reply to this comment if you want to talk more about this stuff):

• https://issarice.com/things-i-wish-i-knew-earlier#learning-how-to-learn
• https://learning.subwiki.org/wiki/Understanding_mathematical_definitions
• https://learning.subwiki.org/wiki/String_replacements_of_code_words
• https://wiki.issarice.com/wiki/Category:Spaced_repetition
• https://www.lesswrong.com/posts/J3Edt2CLcXPHQYSXo/exposition-as-science-some-ideas-for-how-to-make-progress [LW · GW]
• https://issarice.com/anki
• https://issarice.com/reflections-on-five-years-of-conceptual-anki 

Important clarification: when I say that I am doubtful that these observations imply 'consciousness', I am referring to 'phenomenal consciousness'. In other words, an awareness of self and world accompanied by qualia (aka phenomenal experience). What we are seeing does look to me like clear indication of 'access consciousness', but that's insufficient for moral value of the agent. Without the phenomenal experience, the agent can't have valence over its phenomenal experience. In other words, it doesn't have a true internal experience about which to care.

Amanda Askell on AI consciousness.

But suppose it’s possible for an agent has what Block (§4-5, 1995) calls “access consciousness”—it can extract and integrate information from its environment, and access this information when making decisions—and yet lack phenomenal consciousness. Such an agent could have the right architecture for consciousness and behave as if it were conscious, and yet have no phenomenal experiences. On my view, we would be rational to think this agent was phenomenally consciousness based on the strength of evidence available. But if we could somehow see the world from their point of view, we would realize we were mistaken.