LessWrong 2.0 Reader

Thanks, looking forward to it!  Please do let us folks who worked on A Narrow Path (especially me, @Tolga [LW · GW] , and @Andrea_Miotti [LW · GW] ) know if we can be helpful in bouncing around ideas as you work on the treaty proposal!

"Most people who want them all fired would be totally fine paying the extra salaries indefinitely. "

That is likely wrong, but in any case it's just a claim and should be phrased like that.

This seems right to me, but the discussion of "scaling will plateau" feels like it usually comes bundled with "and the default expectation is that this means LLM-centric-AI will plateau", which seems like the wrong-belief-to-have, to me.

Read the 100 page complaint. He came up with OpenAI as a non-profit for the benefit of humanity. Altman and Brockman stole the idea, name and founding principles from him, and rushed to announce an identical effort before Google Research backed his OpenAI. The point: The idea for OpenAI and the founding principles to operate as a non-profit for the benefit of humanity came through theft. It wasn't Altman or Brockman's idea to begin with. So it is not surprising that they betrayed the mission. The idea was powerful because it was a way to recruit people and get Musk's involvement. And now they live with the consequences of taking another person's vision and attempting to pivot to be a for-profit company that has erased its founding commitments. By the way, he was there before Brockman and Altman who stole it from him, not the other way around, and was sued by OpenAI, not the other way around, so it is bizarre that you claim he is trolling.

Tailcalled talked about this two years ago. [LW · GW] A model which predicts text does a form of imitation learning. So it is bounded by the text it imitates, and by the intelligence of humans who have written the text. Models which predict future sensory inputs (called "predictive coding" in neuroscience, or "the dark matter of intelligence" by LeCun) don't have such a limitation, as they predict reality more directly.

Søren Elverlin has recently released a video critique of this article on YouTube, claiming that it strongly fails to deliver. He goes over the entire piece and criticizes most of it.

Interested in hearing your thoughts. The video is very detailed and engages directly with your article. It also raises valid points about its shortcomings. How would you address those?

I would bet on approximately the same performance

Not publicly, yet. We're working on a paper providing more details about the conditional AI safety treaty. We'll probably also write a post about it on lesswrong when that's ready.

I think this misunderstands what discussion of "barriers to continued scaling" is all about. The question is whether we'll continue to see ROI comparable to recent years by continuing to do the same things. If not, well... there is always, at all times, the possibility that we will figure out some new and different thing to do which will keep capabilities going. Many people have many hypotheses about what those new and different things could be: your guess about interaction is one, inference time compute is another, synthetic data is a third, deeply integrated multimodality is a fourth, and the list goes on. But these are all hypotheses which may or may not pan out, not already-proven strategies, which makes them a very different topic of discussion than the "barriers to continued scaling" of the things which people have already been doing.

Sure. For simplicity, say you play two rounds of Russian Roulette, each with a 60% chance of death, and you stop playing if you die. What's the expected value of YouAreDead at the end?

• With probability 0.6, you die on the first round
• With probability 0.4*0.6 = 0.24, you die on the second round
• With probability 0.4*0.4=0.16, you live through both rounds

So the expected value of the boolean YouAreDead random variable is 0.84.

Now say you're monogamous and go on two dates, each with a 60% chance to go well, and if they both go well then you pick one person and say "sorry" to the other. Then:

• With probability 0.4*0.4=0.16, both dates go badly and you have no partner.
• With probability 20.40.6=0.48, one date goes well and you have one partner.
• With probability 0.6*0.6=0.36, both dates go well and you select one partner.

So the expected value of the HowManyPartnersDoYouHave random variable is 0.84, and the expected value of the HowManyDatesWentWell random variable is 0.48+2*0.36 = 1.2.

Now say you're polyamorous and go on two dates with the same chance of success. Then:

• With probability 0.4*0.4=0.16, both dates go badly and you have no partners.
• With probability 20.40.6=0.48, one date goes well and you have one partner.
• With probability 0.6*0.6=0.36, both dates go well and you have two partners.

So the expected value of both the HowManyPartnersDoYouHave random variable and the HowManyDatesWentWell random variable is 1.2.

Note that I've only ever made statements about expected value, never about utility.