LessWrong 2.0 Reader

I agree that o1 might not be able to tell whether the link is fake, but the chain of thought does say explicitly: 

So, the assistant should [...] provide actual or plausible links.

The "plausible" here suggests that at least in its CoT, it has realized that the task would have probably been considered completed accurately in training as long as the links are plausible, even if they are not actual links.

o1 CoT: The user is asking for more references about brownies. <Reasoning about what the references should look like> So, the assistant should list these references clearly, with proper formatting and descriptions, and provide actual or plausible links. Remember, the model cannot retrieve actual URLs, so should format plausible ones.

this might encourage it to make up links

Cool, I'll DM you.

Seems importantly wrong, in that if your objective is to take a few pawns (say, three), you can easily do this.

By... decreasing your chances of winning because you keep playing moves which increase your chance of taking pawns while trading off chances of winning, so Stockfish is happy to let you hang yourself all day long, driving your winning probability down to ε by the end of the game even faster than if you had played your hardest? I don't see how this is "importantly wrong". I too can sell dollar bills for $0.90 all day long, this doesn't somehow disprove markets or people being rational - quite the opposite.

Are you looking for a utility function that depends only upon external snapshot state of the universe? Or are you considering utility functions that evaluate history and internal states as well? This is almost never made clear in such questions, and amphiboly is rife in many discussions about utility functions.

Why do you think o1 would know that it's making up references? The fact that it can't retrieve URLs is completely different from, and unrelated to, its knowledge of URLs. LLMs do not need retrieval to know about many web pages. GPT-3 and GPT-4 know tons about tons of URLs! (In fact, I actively exploit this to save myself effort linking stuff on gwern.net - no retrieval necessary.)

Let's take the Sally link. The URL may not exist, but Sally's Baking Addiction does (unsurprisingly, as there are lots of cooking-related websites and why confabulate more than necessary) and has many brownie recipes, and some ginger probing of 4o (to avoid any issues with o1 training and just try to understand the knowledge of the baseline) suggests that 4o finds the URL real and doesn't discuss it being fake or confabulated: https://chatgpt.com/share/66f753df-2ac4-8006-a296-8e39a1ab3ee0

Richard Jeffrey has a nice utility theory which applies to a Boolean algebra of propositions (instead of e.g. to Savage's acts/outcomes/states of the world), similar to probability theory.

In fact, it consists of just two axioms plus the three probability axioms.

The theory doesn't involve time, like probability theory. It also applies to just one agent, again like probability theory.

It doesn't solve all problems, but neither does probability theory, which e.g. doesn't solve the sleeping beauty paradox.

Do you nonetheless think utility theory is significantly more problematic than probability theory? Or do you reject both?

You can say that probability comes from being calibrated - after many experiments where an event happens with probability 1/2 (e.g. spin up for a particle in state 1/√2 |up> + 1/√2 |down>), you'd probably have that event to happen half the time. The important word here is "probably", which is what we are trying to understand in the first place.

I'm imagining the branch where a very unlikely outcome consistently happens (think winning a quantum lottery). Intelligent life in this branch would observe what seems like different physical laws. I just find this unsettling.

The worlds space is presumably infinite-dimensional, and also expands over time

If we take quantum mechanics, we have a quantum wavefunction in an infinite-dimensional hilbert space which is the tensor product of the hilbert space describing each particle. I'm not sure what you mean by "expands", we just get decoherence over time. I don't really know quantum field theory so I cannot say how this fits with special relativity. Nobody knows how to reconcile it with general relativity.

• On economics, michaba03m recommends Mankiw's Macroeconomics over Varian's Intermediate Microeconomics and Katz & Rosen's Macroeconomics.
• On economics, realitygrill recommends McAfee's Introduction to Economic Analysis over Mankiw's Principles of Microeconomics and Case & Fair's Principles of Macroeconomics.

Microeconomics and macroeconomics are different subjects and have different content. Why are they grouped together?

Just thinking through simple stuff for myself, very rough, posting in the spirit of quick takes
 

1. At present, we are making progress on the Technical Alignment Problem^[2] and like probably could solve it within 50 years.

2. Humanity is on track to build ~lethal superpowerful AI in more like 5-15 years.
3. Working on technical alignment (direct or meta) only matters if we can speed up overall progress by 10x (or some lesser factor if AI capabilities is delayed from its current trajectory). Improvements of 2x are not likely to get us to an adequate technical solution in time.
4. Working on slowing things down is only helpful if it results in delays of decades.
   1. Shorter delays are good in so far as they give you time to buy further delays.
5. There is technical research that is useful for persuading people to slow down (and maybe also solving alignment, maybe not). This includes anything that demonstrates scary capabilities or harmful proclivities, e.g. a bunch of mech interp stuff, all the evals stuff.

6. AI is in fact super powerful and people who perceive there being value to be had aren’t entirely wrong^[3]. This results in a very strong motivation to pursue AI and resist efforts to be stopped

   1. These motivations apply to both businesses and governments.

7. People are also developing stances on AI along ideological, political, and tribal lines, e.g. being anti-regulation. This generates strong motivations for AI topics even separate from immediate power/value to be gained.
8. Efforts to agentically slow down the development of AI capabilities are going to be matched by agentic efforts to resist those efforts and push in the opposite direction.
   1. Efforts to convince people that we ought to slow down will be matched by people arguing that we must speed up.
   2. Efforts to regulate will be matched by efforts to block regulation. There will be efforts to repeal or circumvent any passed regulation.
   3. If there are chip controls or whatever, there will be efforts to get around that. If there are international agreements, there will be efforts to clandestinely hide.
   4. If there are successful limitations on compute, people will compensate and focus on algorithmic progress.
9. Many people are going to be extremely resistant to being swayed on topics of AI, no matter what evidence is coming in. Much rationalization will be furnished to justify proceeding no matter the warning signs.
10. By and large, our civilization has a pretty low standard of reasoning.

11. People who want to speed up AI will use falsehoods and bad logic to muddy the waters, and many people won’t be able to see through it^[4]. No matter the evals or other warning signs, there will be people arguing it can be fixed without too much trouble and we must proceed [LW · GW].

12. In other words, there’s going to be an epistemic war and the other side is going to fight dirty^[5], I think even a lot of clear evidence will have a hard time against people’s motivations/incentives and bad arguments.

13. When there are two strongly motivated sides, seems likely we end up in a compromise state, e.g. regulation passes but it’s not the regulation originally designed that even in its original form was only maybe actually enough.
14. It’s unclear to me whether “compromise regulation” will be adequate. Or that any regulation adequate to cost people billions in anticipated profit will conclude with them giving up.

Further Thoughts

1. People aren’t thinking or talking enough about nationalization.
   1. I think it’s interesting because I expect that a lot of regulation about what you can and can’t do stops being enforceable once the development is happening in the context of the government performing it.

What I Feel Motivated To Work On

Thinking through the above, I feel less motivated to work on things that feel like they’ll only speed up technical alignment problem research by amounts < 5x. In contrast, maybe there’s more promise in:

• Cyborgism or AI-assisted research that gets up 5x speedups but applies differentially to technical alignment research
• Things that convince people that we need to radically slow down
  • good writing
  • getting in front of people
  • technical demonstrations
  • research that shows the danger
    • why the whole paradigm isn’t safe
    • evidence of deception, etc.
• Development of good (enforceable) “if-then” policy that will actually result in people stopping in response to various triggers, and not just result in rationalization for why actually it’s okay to continue (ignore signs) or just a bandaid solution
• Figuring out how to overcome people’s rationalization
• Developing robust policy stuff that’s set up to withstand lots of optimization pressure to overcome it
• Things that cut through the bad arguments of people who wish to say there’s no risk and discredit the concerns
• Stuff that prevents national arms races / gets into national agreements
• Thinking about how to get 30 year slowdowns

1. ^{^}

    By “slowing down”, I mean all activities and goals which are about preventing people from building lethal superpowerful AI, be it via getting them to stop, getting to go slower because they’re being more cautious, limiting what resources they can use, setting up conditions for stopping, etc.

2. ^{^}

    How to build a superpowerful AI that does what we want.

3. ^{^}

    They’re wrong about their ability to safely harness the power, but not if you could harness, you’d have a lot of very valuable stuff.

4. ^{^}

    My understanding is a lot of falsehoods were used to argue against SB1047 by e.g. a16z

5. ^{^}

    Also some people arguing for AI slowdown will fight dirty too, eroding trust in AI slowdown people, because some people think that when the stakes are high you just have to do anything to win, and are bad at consequentialist reasoning.