LessWrong 2.0 Reader

It's not surprising that a lot of people don't want to define physics while believing in physicalism, because properly explaining the equations that describe the physical world would take quite a long time, let alone describing what's actually going on in physics, and it would require a textbook minimum to make this work.

I am not a software engineer, and I've encountered cases where it seems plausible that an engineer has basically stopped putting in work. It can be tough to know for sure for a while even when you notice. But yeah, it shouldn't be able to last for THAT long, but if no one is paying attention?

I've also had jobs where I've had periods with radically different hours worked, and where it would have been very difficult for others to tell which it was for a while if I was trying to hide it, which I wasn't.

I think twice as much time actually spent would have improved decisions substantially, but is tough - everyone is very busy these days, so it would require both a longer working window, and also probably higher compensation for recommenders. At minimum, it would allow a lot more investigations especially of non-connected outsider proposals.

Well that's at least a completely different kind of regulatory failure than the one that was proposed on Twitter. But actually I think this is motivated reasoning on Microsoft's part. If Microsoft wanted, they could also have exported a user API for IDS services, which is a project they are working on now. Microsoft is arguing that they should have a royal monopoly on IDS services, because as it stands high kevel kernel access is necessary in order to provide monitoring and detection services. Microsoft would never have done as good a job on their own if they hadn't faced competition from other companies, which is why everyone uses CrowdStrike in the first place.

If future more capable models are indeed actively resisting their alignment training, and this is happening consistently, that seems like an important update to be making?

Could someone explain to me what this resisting behavior during alignment training looked like in practice?

Did the model outright say "I don't want to do this?", did it produce nonsensical results, did it become deceptive, did it just ... not work?

This claim seems very interesting if true, is there any further information on this?

Ok sorry, I edited it now. Tbc I don't think that this error is out of distribution for errors that I make myself when writing/thinking.

The first idea seems similar to Big World immortality: the concept that due to chaotic inflation, many copies of me exist somewhere, and some of them will not die in any situation. While the copies are the same, the worlds around them could be different, which opens other options for survival: in some worlds, aliens might exist who could save me. The simulation argument can also act as such an anthropic angel, as there will be simulations where I survive. So there can be different observation selection effects that ensure my survival, and it may be difficult to observationally distinguish between them.

Therefore, survival itself is not evidence of MWI, Big World, or simulation. Is that your point?

Regarding the car engineers situation, It is less clear. I know that cars are designed safe, so there is no surprise. Are you suggesting they are anthropic because we are more likely to be driving later in the car evolution timeline when cars are safer?

Do we think there is a non-negligible chance that an AI that doesn't have hidden memory, only English-language CoT, will be able to evade our monitoring and execute a rouge deployment?

Yes, I do think this. I think the situation looks worse if the AI has hidden memory, but I don't think we're either fine if the model doesn't have it or doomed if it does.

I agree that evolutionary arguments are frequently confused and oversimplified, but your argument is proving too much.

[the difference between] AI and genetic code is that genetic code has way less ability to error-correct than basically all AI code, and it's in a weird spot of reliability where random mutations are frequent enough to drive evolution, but not so frequent as to cause organisms to outright collapse within seconds or minutes.

This "weird spot of reliability" is itself an evolved trait, and even with the effects of mutation rate variation between species, the variation within populations is heavily constrained (see Lewontin's paradox of diversity). Even discounting purely genetic/code-based(?) factors, the amount of plasticity (?search) in behaviour is also an evolvable trait (see canalisation) - I think it's likely there are already terms for this within the AI field but it's not obvious to me how best to link the two ideas together. I'm more curious about the value drift evolutionary arguments but I don't see an a priori reason that these ideas don't apply.

It would be good if we could understand the conditions under which greater plasticity/evolvability is selected for, and whether we expect its effects to occur in a timeframe relevant to near-term alignment/safety. 

Another reason is that effective AI architectures can't go through simulated evolution, since that would use up too much compute for training to work (We forget that evolution had at a lower bound 10e46 FLOPs to 10e48 FLOPs to get to humans).

It's not obvious to me that this is a sharp lower-bound, particularly when AI are already receiving the benefits of prior human computation in the form of culture. Human evolution had to achieve the hard part of reifying the world into semantic objects whereas AI has a major head-start. If language is the key idea (as some have argued), then I think there's a decent chance that the lower bound is smaller than this.

How were these discovered? Slow, deliberate thinking, or someone trying some random thing to see what it does and suddenly the AI is a zillion times smarter?