LessWrong 2.0 Reader

A fun lesson on the scientific method: A few days ago, OpenAI announced a rollout of "Advanced Voice Mode". Confused by the messaging, I ran a test to see if their phone app could respond to an interruption:

Nisan: Can you count to 10, and then I'll interrupt you in the middle and see how that works.

ChatGPT 4o: Sure! I'll start counting now: One, two, three, four, fiv—

Nisan: Stop. Interruption.

ChatGPT 4o: How'd the interruption go?

Nisan: Oh, that went great. You stopped at five.

ChatGPT 4o: Awesome! Glad it worked smoothly. Do you want to try another test or experiment?

Nisan: Yeah, let's do another interruption test. This time, please count to 20.

ChatGPT 4o: Got it! Starting now: One, two, three, four, five, six, seven, eight, nine, ten, eleven, twelve, thirt—

Nisan: Stop. Interruption.

ChatGPT 4o: How was that?

Nisan: Wow. [Thinks: It seemed to stop slightly before I interrupted it!]

ChatGPT 4o: Glad that impressed you! Want to try something else?

Nisan: Uh, hold on.

ChatGPT 4o: Take your time!

Nisan: [Generates a random number.] Okay, please count to 30, and I'll interrupt at a random point.

ChatGPT 4o: Alright, starting now: One, two, three, four, five, six, seven, eight, nine, ten, eleven, twelve, thirteen, fourteen, fifteen—

[Nisan interrupts at some point, I forgot exactly where.]

ChatGPT 4o: —sixteen, seventeen, eighteen, nineteen, twenty, twenty-one, twenty-two, twenty-three, twenty-four, twenty-five, twenty-six, twenty-seven, twenty-eight, twenty-nine, thirty. Looks like I made it to the end! Did you mean to interrupt somewhere?

Nisan: Yeah, I did. It must not have worked. Let's try that again. [Generates a random number.] Please count up to 30.

ChatGPT 4o: No problem, let's try again! Starting now: One, two, three, four, five, six, seven, eight, nine, ten, eleven, twelve, thirteen, fourteen, fifteen—

[Nisan interrupts at some point.]

ChatGPT 4o: —sixteen, seventeen, eighteen, nineteen, twenty, twenty-one, twenty-two, twenty-three, twenty-four, twenty-fiv— Did it work this time?

Nisan: Uh, no.

I believe ChatGPT can't hear me when it's talking. But it correctly guessed when I would interrupt on the first two trials!

But if you think taxes are unfair, you can leave the country and pay some different taxes somewhere else instead of going to jail.

It's quite difficult to do that in the US, at least. You pay taxes if you're a citizen, even if you're not a resident, and you're required to pay taxes for the 10 years following your renouncing citizenship.

As far as I know, there's no way for US citizens to leave the US tax regime within a decade.

Petrov Day retro is mostly done, have a bit more coding to do before we publish it, so it's looking like we'll publish it tomorrow. Stay tuned!

"Plausible" is a very ambiguous word. (Bayesianism has been defined as "a logic of plausible inference", but hopefully that doesn't mean Bayesians just confabulate everything.) It can mean "reasonable" for example: "Yeah, Sally's brownie recipe is a reasonable reference to include here, let's go with it." Since 4o doesn't seem to think it's a 'fake' URL in contrast to 'actual' URLs, it's not necessarily a contrast. (It could refer to still other things - like you might not actually know there is a 'brownies' Wikipedia URL, having never bothered to look it up or happened to have stumbled across it, but without retrieving it right this second, you could surely tell me what it would be and that it would be a relevant answer, and so it would be both reasonable and plausible to include.)

I basically agree with this:

People who want to speed up AI will use falsehoods and bad logic to muddy the waters, and many people won’t be able to see through it 

In other words, there’s going to be an epistemic war and the other side is going to fight dirty, I think even a lot of clear evidence will have a hard time against people’s motivations/incentives and bad arguments.

But I'd be more pessimistic than that, in that I honestly think pretty much every side will fight quite dirty in order to gain power over AI, and we already have seen examples of straight up lies and bad faith.

From the anti-regulation side, I remember Martin Casado straight up lying about mechanistic interpretability rendering AI models completely understood and white box, and I'm very sure that mechanistic interpretability cannot do what Martin Casado claimed.

I also remembered a16z lying a lot about SB1047.

From the pro-regulation side, I remembered Zvi incorrectly claiming that Sakana AI did instrumental convergence/recursive self-improvement, and as it turned out, the reality was far more mundane than that:

https://www.lesswrong.com/posts/ppafWk6YCeXYr4XpH/danger-ai-scientist-danger#AtXXgsws5DuP6Jxzx [LW(p) · GW(p)]

Zvi then misrepresented what Apollo actually did, and attempted to claim that o1 was actually deceptively aligned/lying, when it did a capability eval to see if it was capable of lying/deceptively aligned, and straight up lied in claiming that this was proof of Yudkowsky's proposed AI alignment problems being here, and inevitable, which is taken down in 2 comments: 

https://www.lesswrong.com/posts/zuaaqjsN6BucbGhf5/gpt-o1#YRF9mcTFN2Zhne8Le [LW(p) · GW(p)]

https://www.lesswrong.com/posts/zuaaqjsN6BucbGhf5/gpt-o1#AWXuFxjTkH2hASXPx [LW(p) · GW(p)]

Overall, this has made me update in pretty negative directions concerning the epistemics of every side.

There's a core of people who have reasonable epistemics IMO on every side, but they are outnumber and lack the force of those that don't have good epistemics.

The reason I can remain optimistic despite it is that I believe we are progressing faster than that:

1. At present, we are making progress on the Technical Alignment Problem^[2] and like probably could solve it within 50 years.

I think that thankfully, I think we could probably solve it in 5-10 years, primarily because I believe 0 remaining insights are necessary to align AI, and the work that needs to be done is in making large datasets about human values, because AIs are deeply affected by what their data sources are, and thus whoever controls the dataset controls the values of the AI.

Theory: The reason OpenAI seem to not care about getting AGI right any more is because they've privately received explicit signals from the government that they wont be allowed to build AGI. This is pretty likely a-priori, and also makes sense of what we are seeing.

There'd be an automatic conspiracy of reasons to avoid outwardly acknowledging this: 1) To keep the stock up, 2) To avoid accelerating the militarization (closing) of AI and the arms race (a very good reason. If I were Zvi, I would also consider avoiding acknowledging this for this reason, but I'm less famous than zvi, so I get to acknowledge it), 3) To protect other staff from the demotivating effects of knowing this thing, that OpenAI will be reduced to a normal company who will never be allowed to release a truly great iteration of the product.

So instead what you'd see is people in leadership, one by one (as they internalize this), suddenly dropping the safety mission or leaving the company without really explaining why.

I agree that o1 might not be able to tell whether the link is fake, but the chain of thought does say explicitly: 

So, the assistant should [...] provide actual or plausible links.

The "plausible" here suggests that at least in its CoT, it has realized that the task would have probably been considered completed accurately in training as long as the links are plausible, even if they are not actual links.

o1 CoT: The user is asking for more references about brownies. <Reasoning about what the references should look like> So, the assistant should list these references clearly, with proper formatting and descriptions, and provide actual or plausible links. Remember, the model cannot retrieve actual URLs, so should format plausible ones.

this might encourage it to make up links

Cool, I'll DM you.

Seems importantly wrong, in that if your objective is to take a few pawns (say, three), you can easily do this.

By... decreasing your chances of winning because you keep playing moves which increase your chance of taking pawns while trading off chances of winning, so Stockfish is happy to let you hang yourself all day long, driving your winning probability down to ε by the end of the game even faster than if you had played your hardest? I don't see how this is "importantly wrong". I too can sell dollar bills for $0.90 all day long, this doesn't somehow disprove markets or people being rational - quite the opposite.