LessWrong 2.0 Reader

I think your model is a bit simplistic. METR has absolutely influenced the behavior of the big labs, including DeepMind. Even if all impact goes through the big labs, you could have more influence outside of the lab than as one of many employees within. Being the head of a regulatory agency that oversees the labs sets policy in a much more direct way than a mid level exec within the company can.

I'm confused by what you mean that GPT-4o is bad? In my experience it has been stronger than plain GPT-4, especially at more complex stuff. I do physics research and it's the first model that can actually improve the computational efficiency of parts of my code that implement physical models. It has also become more useful for discussing my research, in the sense that it dives deeper into specialized topics, while the previous GPT-4 would just respond in a very handwavy way. 

It's not an entirely unfair characterization.

(Re: Difficult to Parse react on the other comment [LW(p) · GW(p)]
I was confused about relevance of your comment above [LW(p) · GW(p)] on chunky innovations, and it seems to be making some point (for which what it actually says is an argument), but I can't figure out what it is. One clue was that it seems like you might be talking about innovations needed for superintelligence, while I was previously talking about possible absence of need for further innovations to reach autonomous researcher chatbots, an easier target. So I replied with formulating this distinction and some thoughts on the impact and conditions for reaching innovations of both kinds. Possibly the relevance of this was confusing in turn.)

Aren't these different things? Private yes, for profit no. It was private because it's not like it was run by the US government.

The links/graphics are broken btw. Would probably be nice to fix if it's quick.

I think there's a solid case for anyone who supported funding OpenAI being considered at best well intentioned but very naive. I think the idea that we should align and develop superintelligence but, like, good, has always been a blind spot in this community - an obviously flawed but attractive goal, because it dodged the painful choice between extinction risk and abandoning hopes of personally witnessing the singularity or at least a post scarcity world. This is also a case where people's politics probably affected them, because plenty of others would be instinctively distrustful of corporation driven solutions to anything - it's something of a Godzilla Strategy after all, aligning corporations is also an unsolved problem - but those with an above average level of trust in free markets weren't so averse.

Such people don't necessarily have conflicts of interest (though some may, and that's another story) but they at least need to drop the fantasy land stuff and accept harsh reality on this before being of any use.

The piece is unfair towards Bay Area Rationalists, but the critiques of Lumina can stand separate from what the author thinks about LW readers. "Haters gonna occasionally make some valid points" and such. Sometimes people who unfairly dislike you can also make valid critiques. 

I think it's a fair point to note that:

• Lumina have not done any clinical trials
• They circumnavigated the FDA by classifying it as a cosmetic 
• They aren't following best practice guidelines for probiotics (granted actually I don't know how important that is)

On reflection I somewhat endorse pointing the risk out after discovering it, in the spirit of open collaboration, as you did. It was just really frustrating when all my experiments suddenly broke for no apparent reason. But that's mostly on OpenAI for not announcing the change to their API (other than emails sent to some few people). Apologies for grouching in your direction.

There are some conversations about policy & government response taking place. I think there are a few main reasons you don't see them on LessWrong:

1. There really aren't that many serious conversations about AI policy, particularly in future worlds where there is greater concern and political will. Much of the AI governance community focuses on things that are within the current Overton Window.
2. Some conversations take place among people who work for governments & aren't allowed to (or are discouraged from) sharing a lot of their thinking online.
3. LessWrong does not have a history of being a particularly thoughtful place for people to have policy discussions, and a lot of "serious policy people" don't really think LW users will have much to add. 
4. There's a perception that LessWrong has a bit of a libertarian-leaning bias. Some people think LWers are generally kind of anti-government, pro-tech people who are more interested in metastrategies along the lines of "how can me and my smart technical friends save the world" as opposed to "how can governments intervene to prevent the premature development of dangerous technology."

If anyone here is interested in thinking about "40% agreement" scenarios or more broadly interested in how governments should react in worlds where there is greater evidence of risk, feel free to DM me. Some of my current work focuses on the idea of "emergency preparedness"– how we can improve the government's ability to detect & respond to AI-related emergencies.