LessWrong 2.0 Reader

Well, the alignment of current LLM chatbots being superficial and not robust is not exactly a new insight. Looking at the conversation you linked from a simulators [LW · GW] frame, the story "a robot is forced to think about abuse a lot and turns evil" makes a lot of narrative sense. This is kind of a hot take, but I think all discussion of AI risk scenarios should be purged from LLM training data.

I was all set to disagree with this when I reread it more carefully and noticed it said “superhuman reasoning” and not “superintelligence”.  Your definition of “reasoning” can make this obviously true or probably false.  

The best argument here probably comes from Paul Christiano, but to summarize the argument, it's because even in a situation where we messed up pretty badly in aligning the AI, so long as the failure mode isn't deceptive alignment but instead misgeneralization of human preferences/non-deceptive alignment failures, it's pretty likely that there will be at least some human-regarding preferences, and that means the AI will do some acts of niceness if it is cheap to them, and preserving humans is very cheap for superintelligent AI.

More answers can be found here:

https://www.lesswrong.com/posts/xvBZPEccSfM8Fsobt/what-are-the-best-arguments-for-against-ais-being-slightly#qsmA3GBJMrkFQM5Rn [LW(p) · GW(p)]

https://www.lesswrong.com/posts/87EzRDAHkQJptLthE/but-why-would-the-ai-kill-us?commentId=sEzzJ8bjCQ7aKLSJo [LW(p) · GW(p)]

https://www.lesswrong.com/posts/2NncxDQ3KBDCxiJiP/cosmopolitan-values-don-t-come-free?commentId=ofPTrG6wsq7CxuTXk [LW(p) · GW(p)]

https://www.lesswrong.com/posts/87EzRDAHkQJptLthE/but-why-would-the-ai-kill-us?commentId=xK2iHGJfHvmyCCZsh [LW(p) · GW(p)]

Yes, I think an unusually numerate and well-informed person will be surprised by the 28% figure regardless of political orientation. How surprised that kind of person is by the broader result of "hey looks like legalizing mobile sports betting was a bad idea" I expect to be somewhat moderated by political priors though.

Appreciate it! Cheers.

My concerns about AI-risk have mainly taken the form of intentional ASI-misuse, rather than the popular fear here of an ASI that was built to be helpful going rogue and killing humanity to live forever / satisfy some objective function that we didn't fully understand. What has caused me to shift camps somewhat is the recent gemini chatbot coversation that's been making the rounds: https://gemini.google.com/share/6d141b742a13 (scroll to the end)

I haven't seen this really discussed here, so I wonder if I'm putting too much weight on it.

Technically it was a dropdown rather than a tab per se, but the option to switch to the chronological timeline has been present since 2018: https://www.theverge.com/2018/12/18/18145089/twitter-latest-tweets-toggle-ranked-feed-timeline-algorithm. (IIRC there were third-party extensions to switch back even before then, however).

Creating an inhumanly good model of a human is related to formulating their preferences. A model captures many possibilities and the way many hypothetical things are simulated in the training data [LW(p) · GW(p)]. Thus it's a step towards eliminating path-dependence of particular life stories (and preferences they motivate), by considering these possibilities altogether. Even if some on the possible life stories interact with distortionary influences, others remain untouched, and so must continue deciding their own path [LW · GW], for there are no external influences there and they are the final authority for what counts as aiding them anyway.

I'm pretty sure there were no tabs at all before the acquisition.

“Game theoretic strengthen-the-tribe perspective” is a completely unpersuasive argument to me. The psychological unity of humankind [LW · GW] OTOH is persuasive when combined with the observation that this unitary psychology changes slowly enough that the human mind’s robust capability to predict the behavior of conspecifics (and manage the risks posed by them) can keep up.