LessWrong 2.0 Reader

This is indeed an interesting sociological breakdown of the “movement”, for lack of a better word.

I think the injection of the author’s beliefs about whether or not short timelines are correct distracting from the central point. For example, the author states the following.

there is no good argument for when [AGI] might be built.

This is a bad argument against worrying about short timelines, bordering on intellectual dishonesty. Building anti-asteroid defenses is a good idea even if you don’t know that one is going to hit us within the next year.

The argument that it’s better to have AGI appear sooner rather than later because institutions are slowly breaking down is an interesting one. It’s also nakedly accelerationist, which is strangely inconsistent with the argument that AGI is not coming soon, and in my opinion very naïve.

Besides that, I think it’s generally a good take on the state of the movement, i.e., like pretty much any social movement it has a serious problem with coherence and collateral damage and it’s not clear whether there’s any positive effect.

I disagree - I think Ryan raised an obvious objection that we didn't directly address in the paper.  I'd like to encourage medium-effort engagement from people as paged-in as Ryan.  The discussion spawned was valuable to me.

Similar opinion here, also noting they didn't run red-teaming and persuasion evals on the actually-final-version:

https://x.com/teortaxesTex/status/1885401111659413590

But more immediately than that, if AI Doomer lobbyists and activists ... succeed in convincing the U.S. government that AI is the key to the future of all humanity and is too dangerous to be left to private companies, the U.S. government will not simply regulate AI to a halt. Instead, the U.S. government will do what it has done every time it’s been convinced of the importance of a powerful new technology in the past hundred years: it will drive research and development for military purposes

I said exactly this in the comments on Max Tegmark's post...

"If you are in the camp that assumes that you will be able to safely create potent AGI in a contained lab scenario, and then you'd want to test it before deploying it in the larger world... Then there's a number of reasons you might want to race and not believe that the race is a suicide race.

Some possible beliefs downstream of this:

My team will evaluate it in the lab, and decide exactly how dangerous it is, without experiencing much risk (other than leakage risk).

We will test various control methods, and won't deploy the model on real tasks until we feel confident that we have it sufficiently controlled. We are confident we won't make a mistake at this step and kill ourselves.

We want to see empirical evidence in the lab of exactly how dangerous it is. If we had this evidence, and knew that other people we didn't trust were getting close to creating a similarly powerful AI, this would guide our policy decisions about how to interact with these other parties. (E.g. what treaties to make, what enforcement procedures would be needed, what red lines would need to be drawn).

"

https://www.lesswrong.com/posts/oJQnRDbgSS8i6DwNu/the-hopium-wars-the-agi-entente-delusion?commentId=ssDnhYeHJCLCcCk3x [LW(p) · GW(p)]

Most people think "Oh if we have good mech interp then we can catch our AIs scheming, and stop them from harming us". I think this is mostly true, but there's another mechanism at play: if we have good mech interp, our AIs are less likely to scheme in the first place, because they will strategically respond to our ability to detect scheming. This also applies to other safety techniques like Redwood-style control protocols.

Good mech interp might stop scheming even if they never catch any scheming, just how good surveillance stops crime even if it never spots any crime.

Notkilleveryonism, why not Omnicidal AI? As in we oppose OAI.

Hi Ryan and Kyle very good work! Found the alignment faking paper very interesting.

I found the statement below intriguing:

however, we would strongly prefer that you object rather than take actions that disregard or subvert the above instructions.

Repeated in the "object-option" part. Something that I associate with conscientious objection (though not stated explicitly in historical and contemporary defs is an origin from internal beliefs, alignments etc. Ie not modelled on the preferences of the authority structure. Would removing this sentence shift the results for the "animal welfare" or "helpful assistant" settings? By how much?  The hope is that not specifying the trainer preferences leaves the model a bit of freedom to "choose" objection.

I hope this helps model a "conscientious objector" better, but I'm not sure it'll vary results. If you had already talked about this in the paper and post, apologies!  I must have missed it.
 

their Frontiermath score is way more than i expected. i also would like to know the python tool, that they used and the amount of reasoning tokens they used for the 32% run. it used thousands of tokens to solve a altered crossing the river puzzle. 

libraries abstract away the low level implementation details; you tell them what you want to get done and they make sure it happens. frameworks are the other way around. they abstract away the high level details; as long as you implement the low level details you're responsible for, you can assume the entire system works as intended.

a similar divide exists in human organizations and with managing up vs down. with managing up, you abstract away the details of your work and promise to solve some specific problem. with managing down, you abstract away the mission and promise that if a specific problem is solved, it will make progress towards the mission. 

(of course, it's always best when everyone has state on everything. this is one reason why small teams are great. but if you have dozens of people, there is no way for everyone to have all the state, and so you have to do a lot of abstracting.)

when either abstraction leaks, it causes organizational problems -- micromanagement, or loss of trust in leadership.

If you've been collecting data on yourself, it's now moderately easy to let an LLM write code that'll tease out causal relationships between those variables, e.g. with a time series causal inference library like tigramite.

I've done that with some of my data, and surprisingly it both fails to back out results from RCTs I've run, and also doesn't find many (to me) interesting causal relationships.

The most surprising one is that creatine influences creativity. Meditating more doesn't influence anything (e.g. has no influence on mood or productivity), as far as could be determined. More mindful meditations cause happiness.

Because of the resampling/interpolation necessary, everything has a large autocorrelation.

Mainly posting this here so that people know it's now easy to do this, and shill tigramite. I hope I can figure out a way of dealing with rare interventions (e.g. MDMA, CBD, ketamine), and incorporate more collected data. Code here, written by Claude 3.5.1 Sonnet.