LessWrong 2.0 Reader

In other cases, or for other reasons, they might be instead set up to demand results, and evaluate primarily based on results.

Why might it be set up like that? Seems potentially quite irrational. Veering into motivated reasoning territory here imo [LW · GW]

FWIW, my best guess is the document contains fewer errors than having a human copy-paste things and stitch it together. The errors have a different nature to them, and so it makes sense to flag them, but like, I started out with copy-pasting and OCR, and that did not actually have an overall lower error rate.

I'm less confident in this position since I put on a disagree emoji, but my reason is because it's much easier to control an AIs data sources for training than it is for humans, which means it's quite easy in theory (but might be difficult in practice, which worries me) to censor just enough data such that the model doesn't even think that it's likely in a simulation that doesn't add up to normality.

I seem to have gotten a "Why?" on this.

The reason is that checking things yourself is a really, really basic, essential standard of discourse^[1]. Errors propagate, and the only way to avoid them propagating is not to propagate them.

If this was created using some standard LLM UI, it would have come with some boilerplate "don't use this without checking it" warning^[2]. But it was used without checking it... with another "don't use without checking" warning. By whatever logic allows that, the next person should be able to use the material, including quoting or summarizing it, without checking either, so long as they include their own warning. The warnings should be able to keep propagating forever.

... but the real consequences of that are a game of telphone:

1. An error can get propagated until somebody forgets the warning, or just plain doesn't feel like including the warning, and then you have false claims of fact circulating with no warning at all. Or the warning deteriorates into "sources claim that", or "there are rumors that", or something equally vague that can't be checked.
2. Even if the warning doesn't get lost or removed, tracing back to sources gets harder with each step in the chain.
3. Many readers will end up remembering whatever they took out of the material, including that it came from a "careful" source (because, hey, they were careful to remind you to check up on them)... but forget that they were told it hadn't been checked, or underestimate the importance of that.
4. If multiple people propagate an error, people start seeing it in more than one "independent" source, which really makes them start to think it must be true. It can become "common knowledge", at least in some circles, and those circles can be surprisingly large.

That pollution of common knowledge is the big problem.

The pollution tends to be even worse because whatever factoid or quote will often get "simplified", or "summarized", or stripped of context, or "punched up" at each step. That mutation is itself exacerbated by people not checking references, because if you check references at least you'll often end up mutating the version from a step or two back, instead of building even higher on top of the latest round of errors.

All of this is especially likely to happen when "personalities" or politics are involved. And even more likely to happen when people feel a sense of urgency about "getting this out there as soon as possible". Everybody in the chain is going to feel that same sense of urgency.

I have seen situations like that created very intentionally in certain political debates (on multiple different topics, all unrelated to anything Less Wrong generally cares about). You get deep chains of references that don't quite support what they're claimed to support, spawning "widely known facts" that eventually, if you do the work, turn out to be exaggerations of admitted wild guesses from people who really didn't have any information at all. People will even intentionally add links to the chain to give others plausible deniability. I don't think there's anything intentional here, but there's a reason that some people do it intentionally. It works. And you can get away with it if the local culture isn't demanding rigorous care and checking up at every step.

You can also see this sort of thing as an attempt to claim social prestige for a minimal contribution. After all, it would have been possible to just post the link, or post the link and suggest that everybody get their AI to summarize it. But the main issue is that spreading unverified rumors causes widespread epistemic harm.

Sure - I can believe that that's one way a person's internal quorum can be set up. In other cases, or for other reasons, they might be instead set up to demand results, and evaluate primarily based on results. And that's not great or necessarily psychologically healthy, but then the question becomes "why do some people end up one way and other people the other way?" Also, there's the question of just how big/significant the effort was, and thus how big of an effective risk the one predictor took. Be it internal to one person or relevant to a group of humans, a sufficiently grand-scale noble failure will not generally be seen as all that noble (IME).

Maybe, but that also requires that the other group members were (irrationally) failing to consider that the “attempt could've been good even if the luck was bad”. 

In human groups, people often do gain (some) reputation for noble failures (is this wrong?)

I think some international AI governance proposals have some sort of "kum ba yah, we'll all just get along" flavor/tone to them, or some sort of "we should do this because it's best for the world as a whole" vibe. This isn't even Dem-coded so much as it is naive-coded, especially in DC circles.

This inspired me to write a silly dialogue. 

Simplicio enters. An engine rumbles like the thunder of the gods, as Sophistico focuses on ensuring his MAGMA-O1 will go as fast as possible.

Simplicio: "You shouldn't play Chicken."

Sophistico: "Why not?"

Simplicio: "Because you're both worse off?"

Sophistico pats Simplicio's shoulder

Sophistico: "Oh dear, sweet, naive Simplicio! Don't you know that no one cares about what's 'better for everyone?' It's every man out for himself! Really, if you were in charge, Simplicio, you'd be drowned like a bag of mewling kittens."

Simplicio: "Are you serious? You're really telling me that you'd prefer to play a game where you and Galactico hurtle towards each other on tonnes of iron, desperately hoping the other will turn first?"

Sophistico: "Oh Simplicio, don't you understand? If it were up to me, I wouldn't be playing this game. But if I back out or turn first, Galactico gets to call me a Chicken, and say his brain is much larger than mine. Think of the harm that would do to the United Sophist Association! "
 

Simplicio: "Or you could die when you both ram your cars into each other! Think of the harm that would do to you! Think of how Galactico is in the same position as you! "

Sophistico shakes his head sadly. 

Sophistico: "Ah, I see! You must believe steering is a very hard problem. But don't you understand that this is simply a matter of engineering? No matter how close Galactico and I get to the brink, we'll have time to turn before we crash! Sure, there's some minute danger that we might make a mistake in the razor-thin slice between utter safety and certain doom. But the probability of harm is small enough that it doesn't change the calculus."

Simplicio: "You're not getting it. You're race against each other will shift the dynamics of when you'll turn. Each moment in time, you'll be incentivized to go just a little further until there's few enough worlds that that razor-thin slice ain't so thin any more. And your steering won't save from that. It can't. "

Sophistico: "What an argument! There's no way our steering won't be good enough. Look, I can turn away from Galacito's car right now, can't I? And I hardly think we'd push things till so late. We'd be able to turn in time. And moreover, we've never crashed before, so why should this time be any different?"

Simplico: "You've doubled the horsepower of your car and literally tied a rock to the pedal! You're not going to be able to stop in time!"

Sophistico: "Well, of course I have to go faster than last time! USA must be first, you know?"

Simplicio: "OK, you know what? Fine. I'll go talk to Galatico. I'm sure he'll agree not to call you chicken."

Sophistico: "That's the most ridiculous thing I've ever heard. Galactico's ruthless and will do anything to beat me."

Simplicio leaves as Acceleratio helps Sophistico pour in jet fuel to the scramjet engine they've hooked up to O-1.

I agree there are various ways the Face could cause problems via influencing the training of the actual policy. (Both more wild influence that would require clever manipulation and a more mundane reduction of disincentive for problematic reasoning (though I'd note this mundane issue is the core goal of the original proposal!).)

I don't think it is worth getting into all the stuff you're mentioning here, but I think a key crux is that I'm expecting the face to be quite dumb (e.g. literally 3.5 sonnet might suffice).

I agree chess is an extreme example, such that I think that more realistic versions would probably develop instrumental convergence at least in a local sense.

(We already have o1 at least capable of a little instrumental convergence.)

My main substantive claim is that constraining instrumental goals such that the AI doesn't try to take power via long-term methods is very useful for capabilities, and more generally instrumental convergence is an area where there is a positive manifold for both capabilities and alignment, where alignment methods increase capabilities and vice versa.

Really confused as to why this is so heavily downvoted.