LessWrong 2.0 Reader

I think formally, such a circular consequentialist agent should not exist, since running a calculation of X's utility either

• Returns 0 utility, by avoiding self reference.

or

• Runs in an endless loop and throws a stack overflow error, without returning any utility.

However, my guess is that in practice such an agent could exist, if we don't insist on it being perfectly rational.

Instead of running a calculation of X's utility, it has a intuitive guesstimate for X's utility. A "vibe" for how much utility X has.

Over time, it adjusts its guesstimate of X's utility based on whether X helps it acquire other things which have utility. If it discovers that X doesn't achieve anything, it might reduce its guesstimate of X's utility. However if it discovers that X helps it acquire Y which helps it acquire Z, and its guesstimate of Z's utility is high, then it might increase its guesstimate of X's utility.

And it may stay in an equilibrium where it guesses that all of these things have utility, because all of these things help it acquire one another.

I think the reason you value the items in the video game, is because humans have the mesaoptimizer goal of "success," having something under your control grow and improve and be preserved.

Maybe one hope is that the artificial superintelligence will also have a bit of this goal, and place a bit of this value on humanity and what we wish for. Though obviously it can go wrong.

Well yes, but that is just because they are whitelisting it to work with NVLink-72 switches. There is no reason a Hoppper GPU could not interface with NVLink-72 if Nvidia didn't artificially limit it. 

Additionally, by saying
>can't be repeated even with Rubin Ultra NVL576

I think they are indicating there is something else improving besides world size increases, as this improvement would not exist even in 2 more gpu generations when we get 576 (194 gpus) worth of mono-addressable pooled vram, and the giant world / model-head sizes it will enable. 

Interesting. I get where you're coming from for blank slate things or front end. But programming is rarely a blank slate like this. You have to work with existing codebases or esoteric libraries. Even with the context loaded (as well as I can) Cursor with Composer and Claude Sonnet 3.7 code (the CLI tool) have failed pretty miserably for me on simple work-related tasks. As things stand, I always regret using them and wish I wrote the code myself. Maybe this is a context problem that is solved when the models grow to use proper attention across the whole context window rather than using shortcuts or hacks due to memory limitations (this will get better as the newer NVIDIA chips land).

I would love to see an LLM fix a meaningful issue in an open source library. 

"The idea that countries can export and trade-surplus their way to wealth is a fascinating one. They're shipping goods to other countries for free. How then could they prosper more? AFAICT, by outsourcing the task of rewarding and elevating their own most productive citizens."

• tweet by Yudkowsky [LW · GW]

My guess is that he’s referring to the fact that Blackwell offers much larger world sizes than Hopper and this makes LLM training/inference more efficient. Semianalysis has argued something similar here: https://semianalysis.com/2024/12/25/nvidias-christmas-present-gb300-b300-reasoning-inference-amazon-memory-supply-chain

I think "me" is relatively well defined at any instantaneous moment of time.

However, when I try to define "the future me 1 hour later," it is completely subjective who that refers to. If the quantum multiverse (or any cloning machine) creates 100 copies of my current state, and let them evolve in different ways for the next hour, it is subjective which one is the future me, and whose experiences I should anticipate.

There is no objective rule to decide which ones I become. Suppose 99 of my copies has their memories erased one by one, until 90% of their memories are replaced by pigs' memories. Should I anticipate a 99% chance of gradually forgetting everything and becoming a pig, or should I anticipate a 100% chance of remaining as a human?

It's impossible to objectively argue either way. Because if you insist that I do gradually become a pig, then what if that pig then becomes a mouse, and then a fruit fly, and then a bacteria, and then a calculator, and then a rock? Should I anticipate being a rock then? Clearly not since I would be "dead" and hence shouldn't anticipate such an experience, and should only anticipate the experience of my 1 remaining living copy.

But if you insist that I should not anticipate becoming a pig even if 99 of my copies gradually have 90% of their memories replaced by a pigs memories. Then where do you draw the line? What if only 10% of their memories are replaced by a chimpanzee's memories? Or a neanderthal man's memories? Clearly I should continue anticipating their experiences, since they are "still alive" and only experienced a little bit of memory loss.

But there is no objective property in the territory which distinguishes "alive" observers and "dead" observers! Indeed, there is a continuum between living observers and dead observers, e.g. brain damage.

Even if you can objectively define "me" as an observer with the same set of memories M, you have to admit that there is enormous subjectivity deciding who "the me 1 hour later" is. Your decision for which future object you stick the "future me" label on, is a subjective decision. A decision which only affects your map, not the territory.

(This comment is not about the parts which most centrally felt anchored on social reality; see other reply for that. This one is a somewhat-tangential but interesting mini-essay on ontological choices.)

The first major ontological choices were introduced in the previous essay:

1. Thinking of "capability" as a continuous 1-dimensional property of AI
2. Introducing the "capability frontier" as the highest capability level the actor developed/deployed so far
3. Introducing the "safety range" as the highest capability level the actor can safely deploy
4. Introducing three "security factors":
   1. Making the safety range (the happy line) go up
   2. Making the capability frontier (the dangerous line) not go up
   3. Keeping track of where those lines are.

The first choice, treatment of "capability level" as 1-dimensional, is obviously an oversimplification, but a reasonable conceit for a toy model (so long as we remember that it is toy, and treat it appropriately). Given that we treat capability level as 1-dimensional, the notion of "capability frontier" for any given actor immediately follows, and needs no further justification.

The notion of "safety range" is a little more dubious. Safety of an AI obviously depends on a lot of factors besides just the AI's capability. So, there's a potentially very big difference between e.g. the most capable AI a company "could" safely deploy if the company did everything right based on everyone's current best understanding (which no company of more than ~10 people has or ever will do in a novel field), vs the most capable AI the company could safely deploy under realistic assumptions about the company's own internal human coordination capabilities, vs the most capable AI the company can actually-in-real-life aim for and actually-in-real-life not end up dead.

... but let's take a raincheck on clarifying the "safety range" concept and move on.

The safety factors are a much more dubious choice of ontology. Some of the dubiousness:

• If we're making the happy line go up ("safety progress"): who's happy line? Different actors have different lines. If we're making the danger line not go up ("capability restraint"): again, who's danger line? Different actors also have different danger lines.
  • This is important, because humanity's survival depends on everybody else's happy and danger lines, not just one actor's!
• "Safety progress" inherits all the ontological dubiousness of the "safety range".
• If we're keeping track of where the lines are ("risk evaluation"): who is keeping track? Who is doing the analysis, who is consuming it, how does the information get to relevant decision makers, and why do they make their decisions on the basis of that information?
• Why factor apart the levels of the danger and happy lines? These are very much not independent, so it's unclear why it makes sense to think of their levels separately, rather than e.g. looking at their average and difference as the two degrees of freedom, or their average and difference in log space, or the danger line level and the difference, or [...]. There's a lot of ways to parameterize two degrees of freedom, and it's not clear why this parameterization would make more sense than some other.
• On the other hand, factoring apart "risk evaluation" from "safety progress" and "capabilities restraint" does seem like an ontologically reasonable choice: it's the standard factorization of instrumental from epistemic. That standard choice is not always the right way to factor things, but it's at least a choice which has "low burden of proof" in some sense.

What would it look like to justify these ontological choices? In general, ontological justification involves pointing to some kind of pattern in the territory - in this case, either the "territory" of future AI, or the "territory" of AI safety strategy space. For instance, in a very broad class of problems, one can factor apart the epistemic and instrumental aspects of the problem, and resolve all the epistemic parts in a manner totally agnostic to the instrumental parts. That's a pattern in the "territory" of strategy spaces, and that pattern justifies the ontological choice of factoring apart instrumental and epistemic components of a problem.

If one could e.g. argue that the safety range and capability frontier are mostly independent, or that most interventions impact the trajectory of only one of the two, then that would be an ontological justification for factoring the two apart. (Seems false.)

(To be clear: people very often have good intuitions about ontological choices, but don't know how to justify them! I am definitely not saying that one must always explicitly defend ontological choices, or anything like that. But one should, if asked and given time to consider, be able to look at an ontological choice and say what underlying pattern makes that ontological choice sensible.)

The last section felt like it lost contact most severely. It says

What are the main objections to AI for AI safety?

It notably does not say "What are the main ways AI for AI safety might fail?" or "What are the main uncertainties?" or "What are the main bottlenecks to success of AI for AI safety?". It's worded in terms of "objections", and implicitly, it seems we're talking about objections which people make in the current discourse. And looking at the classification in that section ("evaluation failures, differential sabotage, dangerous rogue options") it indeed sounds more like a classification of objections in the current discourse, as opposed to a classification of object-level failure modes from a less-social-reality-loaded distribution of failures.

I do also think the frame in the earlier part of the essay is pretty dubious in some places, but that feels more like object-level ontological troubles and less like it's anchoring too much on social reality. I ended up writing a mini-essay on that which I'll drop in a separate reply.

those conspiracies don't work most of the time "you can only keep a secret between two people, provided one of them is dead".

the personal risk for anyone involved + the human psychological tendency to chat and to have a hard time holding on to immortal secrets mean it's usually irrational for both organisations to do intentional cheating. 

This aged amusingly.