LessWrong 2.0 Reader

It also occurs to me that the causality could go the other way: Ilya and Jan may have timed their departure to coincide with the 4o release for a number of reasons. If they go on to launch a new safety org soon, for example, I'd be more inclined to think that the timing of the two events was a result of Ilya/Jan trying to use the moment to their advantage.

Dragging files around in a GUI is a familiar action that does known things with known consequences. Somewhere on the hard disc (or SSD, or somewhere in the cloud, etc.) there is indeed a "file" which has indeed been "moved" into a "folder", and taking off those quotation marks only requires some background knowledge (which in fact I have) of the lower-level things that are going on and which the GUI presents to me through this visual metaphor.

Some explanations work better than others. The idea that there is stuff out there that gives rise to my perceptions, and which I can act on with predictable results, seems to me the obvious explanation that any other contender will have to do a great deal of work to topple from the plinth. The various philosophical arguments over doctrines such as "idealism", "realism", and so on are more like a musical recreation (see my other comment [LW(p) · GW(p)]) than anything to take seriously as a search for truth. They are hardly the sort of thing that can be right or wrong, and to the extent that they are, they are all wrong.

Ok, that's my personal view of a lot of philosophy, but I'm not the only one.

Oh damn superalignment team has been dissolved.

Every time I start to freak out about AI—be it timelines, risks, or whatever—I come back to this post to get down to earth a bit and avoid making foolish life decisions.

I have a question about this post, and it has to do with the case where both utility and error are heavy tailed:

Where does the expected value converge to if both utility and errors are heavy tailed? Is it 0, infinity, some other number, or does it not converge to any number at all?

Added updates to the post:

Superalignment dissolves.

Leike tweets, including:

I have been disagreeing with OpenAI leadership about the company's core priorities for quite some time, until we finally reached a breaking point.

I believe much more of our bandwidth should be spent getting ready for the next generations of models, on security, monitoring, preparedness, safety, adversarial robustness, (super)alignment, confidentiality, societal impact, and related topics.

These problems are quite hard to get right, and I am concerned we aren't on a trajectory to get there.

Over the past few months my team has been sailing against the wind. Sometimes we were struggling for compute and it was getting harder and harder to get this crucial research done.

Building smarter-than-human machines is an inherently dangerous endeavor. OpenAI is shouldering an enormous responsibility on behalf of all of humanity.

But over the past years, safety culture and processes have taken a backseat to shiny products.

Daniel Kokotajlo tells Vox:

“I joined with substantial hope that OpenAI would rise to the occasion and behave more responsibly as they got closer to achieving AGI. It slowly became clear to many of us that this would not happen,” Kokotajlo told me. “I gradually lost trust in OpenAI leadership and their ability to responsibly handle AGI, so I quit.” 

Kelsey Piper says:

I have seen the extremely restrictive off-boarding agreement that contains nondisclosure and non-disparagement provisions former OpenAI employees are subject to. It forbids them, for the rest of their lives, from criticizing their former employer. Even acknowledging that the NDA exists is a violation of it.

https://x.com/janleike/status/1791498174659715494?s=46&t=lZJAHzXMXI1MgQuyBgEhgA

Leike explains his decisions.

We ran into a hardware shortage during a period of time where there was no pause, which is evidence that the hardware manufacturer was behaving conservatively.

Alternative hypothesis, there are physical limits on how fast you can build things.

Also, NVIDIA currently has a monopoly on "decent AI accelerator you can actually buy".  Part of the "shortage" is just the standard economic result that a monopoly produces less of something to increase profits.  

This monopoly will not last forever, so in that sense we are currently in hardware "underhang".
 

This and the rest of your comment seems to have ignored the rest of my post (see: multiple inputs to progress, all of which seem sensitive to "demand"

Nvidia doesn't just make AGI accelerators.  They are are video game graphics card company. 

And even if we pause large training runs, demand for inference of existing models will continue to increase.

If you think my model of how inputs to capabilities progress are sensitive to demand for those inputs from AGI labs is wrong, then please argue so directly, or explain how your proposed scenario is compatible with it.

This is me arguing directly.  

The model "all demand for hardware is driven by a handful of labs training cutting edge models" is completely implausible. It doesn't explain how we got the hardware in the first place (video games) and it ignores the fact that there exist uses for AI acceleration hardware other than training cutting-edge models.

My Manifold market on Collin Burns, lead author of the weak-to-strong generalization paper

If they ban it, then this would count as "selling it on the street corners in retaliation". For now, I think Lumina should get the profit for the research they did. 

Cases of people doing mouth-to-mouth will also make it easier for the American Dental Association to smear Lumina, which might even shift the Overton window enough to accidentally turn public opinion against similar products in the future (e.g. like what happened with cryonics, literally billions of people died as a direct result of early marketing/political failures [? · GW]).