LessWrong 2.0 Reader

[I]s it doing something actively bad, or is it just a waste of resources?

Just a waste of resources, more or less. But I assume a utility function maximizer doesn't draw a qualitative difference between these two categories - it's just a difference in magnitude.

To go with your example, from the perspective of humanity, I wouldn't actually say humanity would view the paperclip maximizer as neutral w/r/t themselves. Those are still planets we might, someday, far off, want for our own purposes.

If the AIs [...] don't assume the probability of victory to be significantly higher than 50%, they could just give up on the other half of universe, and e.g. burn up the resources along the boundary, to make it more difficult to travel to each other.

I don't think two AIs meeting at the boundary I'm describing would be able to reliably update their priors away from 50% quickly, for two reasons. (I am assuming they start there, a la prior of indifference.)

One is physics-based. When we're talking about scales this large, the speed of light itself can become a barrier to discovering enough information about your opponent to know what their true capabilities are. 

The second is game theoretic. We're thinking about two superintelligences meeting - they're probably going to try all kinds of tactics to convince each other that the other side has the success probability they want them to think. I think the net effect of all those spy games would be to add a lot of uncertainty about any kind of updating, in one direction or the other.

burn up the resources along the boundary, to make it more difficult to travel to each other

(Physically travel, sure. But there's no reason they can't just beam hostile radio signals at each other all day long. There isn't a lot of attenuation in empty space, and it's hard to respond to something travelling at epsilon-less-than light speed. If you can make a decent guess at what the architecture of the other side's computronium is, you can hack it with those, and if you can remotely take control of the other side's compute, maybe you don't need to physically travel there at all anymore.)

People interested in approaches to solving reward hacking which don't depend on supervision (and thus might scale to superintelligence) should consider looking at our earlier work on measurement tampering [AF · GW]. I don't particularly reccomend the benchmark atm, but the discussion in the blog post might be interesting. This type of setting makes some assumptions about the type and structure of the reward hacking, but I think it will be hard to resolve reward hacking in a principled way without supervision without assumptions similar to these.

Yes, this seems the most likely. His prompt says "Hivemind provides an optimized all-reduce algorithm designed for execution on a pool of poorly connected workers"

The "Hamas" feature is slightly triggered by the words "execution" "of" "poorly" "workers," as well as the words "decentralized network" (which also describes Hamas), "checkpoint," and maybe "distributed training."

If the LLM was operating normally, the "Hamas" feature should get buried by various "distributed computing" features.

But since OpenAI trained it to respond extremely consistently about Hamas prompts, it is absurdly oversensitive to the "Hamas" feature.

I always find the use of "X% of the vote" in US elections to make some general point about overall acceptability or representation of the general public problematic. I agree it's a true statement but leaves out the important aspect of turn out for the vote. 

I have to wonder if, particularly the USA, would not be quite as divided if all the reporting provided percentage of vote of the eligible voting population rather than the percentage of votes cast. I think there is a big problem with just ignoring the non-vote information the is present (or expecting anyone to look it up and make the adjustments on their own).

But I agree, I'm not too sure just where electoral systems fall into this question of AGI/ASI first emerging under ether the USA or CCP.

I think there's a possibility for ui people to make progress on the reputation tracking problem by virtue of tight feedback loops relative to people thinking more abstractly about it.

Are there particular reputation-tracking-problems you're thinking of? (I'm sure there are some somewhere, but I'm looking to get more specific)

I'm working on a poweruser LLM interface but honestly it's not going to be that much better than Harpa AI or Sider.

I work mostly as a distiller (of xrisk-relevant topics). I try to understand some big complex thing, package it up all nice, and distribute it. The "distribute it" step is something society has already found a lot of good tech for. The other two steps, not so much.

Loom [LW · GW] is lovely in the times I've used it. I would love to see more work done on things like this, things that enhance my intelligence while keeping me very tightly in the loop. Other things in this vein include:

• memory augmentation beyond just taking notes (+Anki where appropriate). I'm both interested in working memory and long-term memory, but moreso in the former.
• For digesting complex pieces, I'd like something better than a chat window for interactively getting up to my desired level of familiarity with some existing work. When I'm digesting the paper, do I want the one-sentence summary, the abstract, or to have a better understanding than the researchers behind it? NotebookLM is sort of doing this but I've found it lacking for this task for UI reasons (I also wish that it would automatically bring in other relevant sources).

小句子很好。为什么？清晰。我的句子又长又抽象。不清楚。现在简短清晰了。

很好。思路清晰。为什么？简单需要深刻。

Would I be incorrect in reading into the post that, at least looked at through the lens of geopolitical status and power, one might not view the post WWII international order than promoted a lot of investment in the developing countries economic and health was planting the seeds of the West's vision of the modern world order?

Or do you see the current state of things as occurring independently of relative population changes?

Oops meant aistudio.google.com

We can get compute outside of the labs. If grantmakers, government, donated compute from service providers, etc are willing to make a group effort and take action, we could get an additional several millions in compute spent directly towards automated safety. An org that works towards this will be in a position to absorb the money that is currently inside the war chests.

This is an ambitious project that makes it incredibly easy to absorb enormous amounts of funding directly for safety research.

There are enough people who work in AI safety who want to go work at the big labs. I personally do not need or want to do this. Others will try by default, so I’m personally less inclined. Anthropic has a team working on this and they will keep working on it (I hope it works and they share the safety outputs!).

What we need is agentic people who can make things happen on the outside.

I think we have access to frontier models early enough and our current bottleneck to get this stuff off the ground is not the next frontier model (though obviously this helps), but literally setting up all of the infrastructure/scaffolding to even make use of current models. This could take over 2 years to set everything up. We can use current models to make progress on automating research, but it’s even better if we set everything up to leverage the next models that will drop in 6 months and get a bigger jump in automated safety research than what we get from the raw model (maybe even better than what the labs have as a scaffold).

I believe that a conscious group effort in leveraging AI agents for safety research, it could allow us to make current models as good (or better) than the next generations models. Therefore, all outside orgs could have access to automated safety researchers that are potentially even better than the lab’s safety researchers due to the difference in scaffold (even if they have a generally better raw model).