LessWrong 2.0 Reader

Try to apply ability to stop to the process of not-doing

amazing. i just need to turn my tendency towards deconstruction inwards. :p

In my hypothesis, scrolling is a bad way to rest, because it actually eats up a lot of cognitive power

agreed. i deactivated my twitter last year, but alas, i keep checking lesswrong

Ok, I think you might be lumping this under "international governance", but I feel like you are severely underestimating the potential of open source AI. (Or off-brand secret projects based on open source AI). Right now, the big labs have a lead, and that lead is expected to grow temporarily in the short term since they have been scaling their compute to ranges out of reach to B-class players.

But... What happens when algorithmic innovation gets accelerated? It's already somewhat accelerated now. How secret are you going to manage to keep the findings about potential algorithmic improvements? It is a LOT easier for one untrustworthy employee to leak a concept about how to approach model architectures more efficiently than it is for them to steal the existing model weights.

Also, what do you do if existing open weights models at the time of international lockdowns on training runs are already sufficiently strong to be dangerous with only tweaks to the fine-tuning and RL protocols?

Is this so far outside your Overton window that you don't feel it's worth discussing?

Deepseek-V3 has 37B active params and 671 total params. Thats a ratio of about 18.14:1. What if someone made an MoE model using Llama 405B, with 405B active params and 7.4 trillion total params? Ehat if a new pre-training run wasn't needed, just RL fine-tuning?

What about Llama 4, that is in progress? Will your international governance prevent that from being released?

My main issue with this post is that it seems substantially concerned with losing the ability for humans to achieve significant levels of wealth and power relative to each other, which I agree is important for avoiding a calcified ruling class (which tends to go poorly for a society, historically). But it should be viewed as a transitional concern as we look towards building a society where radical wealth disparities (critically, here defined as the power of the wealthy to effectively incentivize the less wealthy to endure unwanted experiences or circumstances, e.g., long work weeks, poorer nutrition or education, long commutes). 

Human ambition isn't worth human poverty except exactly to the degree that it may be necessary to eliminate suffering. 

I think the concern here is twofold:

1. Once a model is deceptive at one point, even if this happens stochastically, it may continue in its deception deterministically.
2. We can't rely on future models being as stochastic w.r.t the things we care about, e.g. scheming behaviour.

Regarding 2, consider the trend towards determinicity we see for the probability that GPT-N will output a grammatically correct sentence. For GPT-1 this was low, and it has trended upwards towards determinicity with newer releases. We're seeing a similar trend for scheming behaviour (though hopefully we can buck this trend with alignment techniques).

It doesn't change the picture a lot because the proposal for preventing misaligned goals from arising via this mechanism was to try and get control over when the AI does/doesn't step back, in order to allow it in the capability-critical cases but disallow it in the dangerous cases. This argument means you'll have more attempts at dangerous stepping back that you have to catch, but doesn't break the strategy.

The strategy does break if when we do this blocking, the AI piles on more and more effort trying to unblock it until it either succeeds or is rendered useless for anything else. There being more baseline attempts probably raises the chance of that or some other problem that makes prolonged censorship while maintaining capabilities impossible. But again, just makes it harder, doesn't break it.

I don't think you need to have that pile-on property to be useful. Consider MTTR(n), the mean time an LLM takes to realize it's made a mistake, parameterized by how far up the stack the mistake was. By default you'll want to have short MTTR for all n. But if you can get your MTTR short enough for small n, you can afford to have MTTR long for large n. Basically, this agent tends to get stuck/rabbit-hole/nerd-snipe but only when the mistake that caused it to get stuck was made a long time ago.

Imagine a capabilities scheme where you train MTTR using synthetic data with an explicit stack and intentionally introduced mistakes. If you're worried about this destabilization threat model, there's a pretty clear recommendation: only train for small-n MTTR, treat large-n MTTR as a dangerous capability, and you pay some alignment tax in the form of inefficient MTTR training and occasionally rebooting your agent when it does get stuck in a non dangerous case.

Figured I should get back to this comment but unfortunately the chewing continues. Hoping to get a short post out soon with my all things considered thoughts on whether this direction has any legs

also see created already in motion [LW · GW]. this applies to more than just priors.

Another minor inconvenience is that it’s not terribly easy to search my shortform. Ctrl + F works reasonably well when I’m on laptop. On mobile the current best option is LW search + filter by comments. This is a little bit more friction than I would like but it’s tolerable I guess

But you see, that's exactly the point!

What is the eval trying to measure?

• if you are measuring how safe a model is to deploy as-is behind a closed API, then fine. No fine-tuning evals needed (as long as the API doesn't offer fine-tuning, in which case you need fine-tuning evals that take the API protections into account.)

• if you are measuring how dangerous the model weights would be if they were stolen and completely under the control of bad actors... Obviously you need fine-tuning evals! Why wouldn't you expect the bad actors to fine-tune on the task they wanted the model to do well on?!

• if you are measuring how dangerous the model weights will be for a model you intend to publish the weights openly for, same deal. If you wait to do this testing until after you've made the weights public, or you make arguments like "we trust the open source community will let us know if they discover anything hazardous". That's an argument that makes some sense with an open source code library. Users notice a flaw, they report it, the maintainers patch the flaw, users update to latest version, bug is gone. That is not a good model for how problematic capabilities discovered in open-weight models could be handled.

One of the missing considerations is that crime is done mostly by young people, and the rate of crimes goes down the older you get.

A lot of this IMO is that the impulsiveness/risk-taking behavior of crimes decreases a lot with age, but the empirical fact of crime going down with age, especially reoffending is a big reason why locking people up for life is less good than Ben Pace said, because the reoffending rate goes down with age.

Kim Stanley Robinson seems to fit this too:

In some ways, Robinson’s path as a science fiction writer has followed a strange trajectory. He made his name writing about humanity’s far-flung future, with visionary works about the colonization of Mars (“The Mars Trilogy”), interstellar, intergenerational voyages into deep space (“Aurora”), and humanity’s expansion into the far reaches of the solar system (“2312”). But recently, he’s been circling closer to earth, and to the current crisis of catastrophic warming.

Futuristic stories about space exploration feel irrelevant to him now, Robinson said. He’s grown skeptical that humanity’s future lies in the stars, and dismissive of tech billionaires’ ambitions to explore space, even as he acknowledged, “I’m partially responsible for that fantasy.”

In his more recent novels — works like “New York 2140,” an oddly uplifting climate change novel that takes place after New York City is partly submerged by rising tides, and “Red Moon,” set in a lunar city in 2047 — he has traveled back in time, toward the present. Two years ago, he published “The Ministry for the Future,” which opens in 2025 and unfolds over the next few decades, as the world reels from floods, heat waves, and mounting ecological disasters, and an international ministry is created to save the planet.