LessWrong 2.0 Reader

Entropy production partially solves the Strawberry Problem:

Change in entropy production per second (against the counterfactual of not acting) is potentially an objectively measurable quantity that can be used either in conjunction with other parameters specifying a goal to prevent unexpected behaviour.

Rob Bensinger gives Yudkowsky's "Strawberry Problem [LW · GW]" as follows:

How would you get an AI system to do some very modest concrete action requiring extremely high levels of intelligence, such as building two strawberries that are completely identical at the cellular level, without causing anything weird or disruptive to happen?

I understand the crux of this issue to be that it is exceptionally difficult for humans to construct a finite list of caveats or safety guardrails that we can be confident would withstand the optimisation pressure of a super intelligence doing its best to solve this task "optimally". Without care, any measure chosen is Goodharted into uselessness and the most likely outcome is extinction.  

Specifying that the predicted change in entropy production per second of the local region must remain within some $\delta$ of the counterfactual in which the AGI does not act at all automatically excludes almost all unexpected strategies that involves high levels of optimisation.

I conjecture that the entropy production "budget" needed for an agent to perform economically useful tasks is well below the amount needed to cause an existential disaster. 

Another application, directly monitoring the entropy production of an agent engaged in a generalised search [LW · GW] upper bounds the number of iterations of that search (and hence the optimisation pressure). This bound appears to be independent of the technological implementation of the search. ^[1]

1. ^{^}

   On a less optimistic note, this bound is many orders of magnitude above the efficiency of today's computers.

Footnote to table cells D18 and D19:

My reading of Anthropic's ARA threshold, which nobody has yet contradicted:

1. The RSP defines/operationalizes 50% of ARA tasks (10% of the time) as a sufficient condition for ASL-3. (Sidenote: I think the literal reading is that this is an ASL-3 definition, but I assume it's supposed to be an operationalization of the safety buffer, 6x below ASL-3.)
2. The May RSP evals report suggests 50% of ARA tasks (10% of the time) is merely a yellow line. (Pages 2 and 6, plus page 4 says "ARA Yellow Lines are clearly defined in the RSP" but the RSP's ARA threshold was not just a yellow line.)
3. This is not kosher; Anthropic needs to formally amend the RSP to [raise the threshold / make the old threshold no longer binding].

(It's totally plausible that the old threshold was too low, but the solution is to raise it officially, not pretend that it's just a yellow line.)

(The forthcoming RSP update will make old thresholds moot; I'm just concerned that Anthropic ignored the RSP in the past.)

(Anthropic didn't cross the threshold and so didn't violate the RSP — it just ignored the RSP by implying that if it crossed the threshold it wouldn't necessarily respond as required by the RSP.)

One of the highest quality-of-life increases I've ever experienced is when I stopped listening to music with sad lyrics. Crazy how long it took me to realize it was lowering my mood in a persistent, motivation-sapping way.

I want to run one of @Ben Pace [LW · GW]'s polls at the bottom here. Please could people put statements that they might want to agree or disagree with relating to this essay as comments here. Some starters:

• If the UK wants to grow then it would do well to give energy production, housing and infrastructure a higher priority
• France is able to be dysfunctional and still wealthy because it gets the basics of housing, energy and infrastructure right
• The UK Town and Planning act was probably very damaging
• If the UK wants more growth it should build more housing where people want to live
• I think that cities would generally grow more if they had more people in them

This assumes a task-first model of agency, whereas one could instead develop a resource-first model of agency.

If an AI learns to segment the universe into developable resources and important targets that the resources could be propagated into modifying, then the AI could simply remain under human control.

The conventional reason for why this cannot work is that the relevant theories of resource-development agency (as opposed to task-solution agency) haven't been developed, but that is looking less and less important with current developments in AI. Like yes, current AIs can sort of do task-solution in environments like CTF where that is less relevant, but for serious and dangerous tasks, more effort will likely go into resource-development agency than task-solution agency because resource-development agency is safer. And resource-development agency provides a natural sort of impact measure etc. that restrains whatever fragments of task-solution agency develop in order to complement the resource-development agency.

(And an important aspect of resource-development agency is that you don't really need a complete theory, you can just develop each part separately, because there's only so many resources and so many interesting targets to develop them towards. Like think stuff like metabolism or the interplanetary transport network, where there's sort of a small canonical solutionspace that is very critical. Really all of reality is like that.)

The actual reason resource-development agency doesn't work is security. In order to sufficiently quickly and sufficiently dynamically respond to adversarial threats, the AIs cannot wait for painfully slow humans to make decisions about what to do. So what constitutes a threat and what are acceptable ways of neutralizing them needs to be decided ahead of time, and it needs to be sufficiently aggressive against threats that the security-provider doesn't get destroyed by something bad while being sufficiently open-ended that the security-provider doesn't cause permanent stagnation of the world.

"Workers regularly trade with billionaires and earn more than $77 in wages, despite vast differences in wealth."

Yes, because the worker has something the billionaire wants (their labor) and so is able to sell it. Yudkowsky's point about trying to sell an Oreo for $77 is that a billionaire isn't automatically going to want to buy something off you if they don't care about it (and neither would an ASI).

"I'm simply arguing against the point that smart AIs will automatically turn violent and steal from agents who are less smart than they are, unless they're value aligned. This is a claim that I don't think has been established with any reasonable degree of rigor."

I completely agree but I'm not sure anyone is arguing that smart AIs would immediately turn violent unless it was in their strategic interest.

I don't know how the survey was done, but one silver lining is that if the respondents were asked those questions in the same order they are shown then the last question was given last in that set. So the people being surveyed are to some extent being "brought towards it" by the preceding questions. See this Yes Minister clip (although the survey above is much less dramatic example, and they published all the questions, so they are being legitimate): 

You could probably cook up a set of preceding questions that would have delivered the opposite result. Something like: "Do you think Germans in WW2 should have resisted/criticised their government more?", following with questions about a non-specific country's people criticising its government during a non-specific war, then going to that final question. 

While 2024 SoTA models are not capable of autonomously optimizing the world, they are really smart, perhaps 1/2 or 2/3 of the way there, and already beginning to make big impacts on the economy. As I said in response to your original post, because we don't have 100% confidence in the coherence arguments, we should take observations about the coherence level of 2024 systems as evidence about how coherent the 203X autonomous corporations will need to be. Evidence that 2024 systems are not dangerous is both evidence that they are not AGI and evidence that AGI need not be dangerous.

I would agree with you if the coherence arguments were specifically about autonomously optimizing the world and not about autonomously optimizing a Go game or writing 100-line programs, but this doesn't seem to be the case.

the mathematical noose [LW · GW] around them is slowly tightening [LW · GW]

This is just a conjecture, and there has not really been significant progress on the agent-like structure conjecture. I don't think it's fair to say we're making good progress on a proof.

This might be fine if proving things about the internal structure of an agent is overkill and we just care about behavior? In this world what the believers in coherence really need to show is that almost all agents getting sufficiently high performance on sufficiently hard tasks score high on some metric of coherence. Then for the argument to carry through you need to show they are also high on some metric of incorrigibility, or fragile to value misspecification. None of the classic coherence results quite hit this.

However AFAIK @Jeremy Gillen [LW · GW] does not think this is true, and Eliezer has historically and recently [LW · GW] made the argument that EU maximization is simple and natural. So maybe you do need this argument that an EU maximization algorithm is simpler than other algorithms, which seems like it needs some clever way to formalize it, because proving things about the space of all simple programs seems too hard.

Yeah, I agree with the general point (don't have strong opinion about chaos theory at the moment).

First, negative results are really, really important. Mostly because they let you not lose your time trying to do something impossible, and sometimes they actually point you toward an answer. In general, conservation laws in physics have this role.  And knowing what is undecidable is really important in formal methods, where the trick is generally to simplify what you want or the expressive power of your programs in order to sidestep it.

Then, they are indeed quite hard to prove, at least in non-trivial cases. Conservations laws are the results of literally centuries of reframing of classical mechanics and reduction, leading to seeing the importance of energy and potential in unifying everything in physics. Undecidability is the result of 60 years of metamathetical work trying to clean formalisms enough to be able to study these kind of properties.

Hmm I don't recall no longer noticing things that I care about while on them. Of course, since I was less anxious, then having less anxiety also meant noticing less anxiety. But I think that either the anxiety just felt content-less and not associated with anything that I cared about, or to the extent that it was tracking things I cared about, I was able to track those even without it.