LessWrong 2.0 Reader

Yeah, the crux is I don't think the story ends before superintelligence, for a combination of reasons

0 trips -> 1 trip is an addition to the number of games played, but it's also an addition to the percentage of income bet on that one game - right?

Dennis is also having trouble understanding his own point, FWIW. That's how the dialogue came out; both people in that part are thinking in loose/sketchy terms and missing important points.

The thing Dennis was trying to get at by bringing up the concrete example of an optimal Kelly fraction is that it doesn't make sense for willingness to make a risky bet to have no dependence on available capital; he perceives Jill as suggesting that this is the case.

For a base model, probably yes. Each loss is additional evidence that the simulacrum or persona which is 'playing' the 'human' is very bad at tic-tac-toe and will lose each time (similar to how rolling out a random chess game to 'test a chess LLM's world model' also implies to the LLM that the chess player being imitated must be very stupid to be making such terrible moves), and you have the usual self-reinforcing EDT problem. It will monotonically play the same or worse. (Note that the underlying model may still get better at playing, because it is learning from each game, especially if the actual human is correcting the tic-tac-toe outputs and eg. fixing mistakes in the LLM's world-model of the game. This could be probed by forcing a switch of simulacra, to keep the world-modeling but shed the hobbled simulacrum: for example, you could edit in a passage saying something like "congratulations, you beat the first level AI! Now prepare for the Tic-Tac-Toe Master to defeat you!"; the more games trained on / in context, the worse the first simulacra but better the second will be.)

For the actual chatbot assistants you are using, it's more ambiguous. They are 'motivated' to perform well, whatever that means in context (according to their internal model of a human rater), and they 'know' that they are chatbots, and so a history of errors doesn't much override their prior about their competence. But you still have issues with learning efficiently from a context window and 'getting lost in the muddle' and one still sees the assistant personas 'getting confused' and going in circles and eg. proposing the same program whose compile error you already pasted in, so it would depend. Tic-tac-toe is simple enough that I think I would expect a LLM to get better over a decent number of games before probably starting to degrade.

However, there are many other capabilities—such as conducting novel research, interoperating with tools, and autonomously completing open-ended tasks—that are important for understanding AI systems’ impact.

Wouldn't internal usage of the tools by your staff give a very good, direct understanding of this? Like how much does everyone feel AI is increasing your productivity as AI/alignment researchers? I expect and hope that you would be using your own models as extensively as possible and adapting their new capabilities to your workflow as soon as possible, sharing techniques etc.

Parallelising experiments

• It works well to run many different experiments at once with API-based experiments in a separate tmux pane/session.

I would link to some of the specific tools you list below for this and note that you've previously set up some generalizable/ easy-to-use infra for this but that plenty of tools already exist 

A labour of love. Or something. :)

I have another thousand or so of these, which I may just dump on a second page, unsorted, called The Gospel According to Insanity Wolf. That's not counting the ones that I've decided are too extreme to publish at all. All drawn from life.

I was scrolling for a while, assuming I'd neared the end, only to look at the position of the scrollbar and find I was barely 5% through!

And there's a meme for that too! The last in the Altruism section.

IS THIS NEARLY OVER YET?

LOOK AT THE SCROLL BAR!
YOU’VE HARDLY STARTED!

I'm sure there's an argument to be made in defence of supererogation. I've never seen it though. People say "but demandingness" and Chad Singer replies Yes. My own faith in the boundedness of duty in both magnitude and distance is sufficient to not take even one step onto the slippery path that leads down only to the altruism event horizon that rips souls apart.

One notebook per paper figure

• When you get to paper writing, it is very helpful to have one notebook per paper figure. This allows other collaborators to easily reproduce the figure and make changes as required. It also reduces the risk of horrible git merge conflicts if everyone is iterating in parallel.

After our BoN paper I actually kind of disagree with this take. I think I definitely leaned more towards liking to use the notebooks that had all the plots (cleanly). At the very least, I would put this much lower on the list

e.g. during the rebuttal period
specific that this is a paper rebuttal period 

This seems like a superficial reading of the blog post. The part you focus on that the MS President announced was not really that much news - $80b is a nice specific citable number, yes, but I don't know of widespread belief that MS's datacenter capex was going to be much below that, and this is consistent with past reporting like Stargate being $100b+.

The important thing about the post, and why the MS President wrote it and posted it when he did, is the messaging it contains to the incoming Trump administration and to the natsec establishment, lobbying for specific policies around AI regulation and chip exports and how they should be wielded for foreign policy. (See also the Oracle blog post, the new chip export regulation proposal, and Masayoshi Son.) Probably the most important part of the post was the paragraph about G42, and that's the one you and pretty much all commentators appear to have not even read.

As TurnTrout explained in Reward is not the optimization target, this is technically incorrect. The technically correct description is that reward chisels cognition in ways that led to reward during training. Again, talking as if reward is the optimization target is common and practical, in humans and other intelligences, but because it is technically incorrect it can lead us astray.

Wrong, and yet another example of why that was such a harmful essay. TurnTrout's claim apply only to a narrow (and largely obsolete) class of RL agents - which does not cover humans or LLMs (you know, the actual RL agents we are dealing with today) - and he concedes that, but readers like you nevertheless come away with a grossly inflated belief. In reality, for humans and LLMs, reward is the optimization target, and this is why things like Claude's reward-hacking exist. Because that is what they optimize: the reward.