LessWrong 2.0 Reader

I'd be happy to put the opening bunch of paragraphs. I was feeling reluctant to cross-post because I often update my articles as I learn more about a topic, and I don't want to keep multiple versions in sync (especially for a lower priority article).

btw, I was getting ready for an appointment earlier, so I had only skimmed this until now.  Thank you for doing this and sharing it.  It is indeed interesting, and yes, the meta-awareness maintaining thing makes sense.  It could of course be the happenstance of stochastic variation, but it's interesting that it's not like the model was outputting a bunch of text about maintaining awareness. If it wasn't actually doing anything, except for pre-emptively outputting text that spoke of awareness, then token prediction would just have the output be just as reliable. The fact that it aligned with the self-reported difficulty suggests that it's doing something at the very least.  

I just realized that's what you just said in rereading your concluding paragraph, but i was just coming to the same conclusion in real-time. Genuinely excited that someone else is engaging with this and tackling it from a different angle.  

It's not about which part "equilibrates" faster (with what?) but about the relative strength of the shifts.

If demand for socks quadruples overnight for some exogenous reason, then there will be some period of time when socks are more expensive. Days, weeks, months, I don’t know. But there’s a lead time on building new sock factories (or hiring more workers, or retooling production lines, or whatever it is). In the longer term, presumably socks would wind up at a similar price as today—there’s plenty of cotton. The supply curve can shift, but can't shift instantaneously.

So that’s what I mean by “equilibration”—The demand curve shifts, and then we’re “out of equilibrium”, in the sense that there’s a highly-profitable opportunity (to manufacture more socks), but nobody is exploiting that opportunity yet (because it takes a few weeks or months). Eventually people do build the factories, and now the supply curve has shifted to its new stable home, and the price of socks is its normal inexpensive price as usual.

Hmm, I wonder if you’re assuming that “equilibration” = “price moving to match immediate supply and demand”, whereas I’m using the term “equilibration” more broadly (“some process reaching equilibrium”). I’m open-minded to rewording things. I do want this post to use conventional economics language as much as possible, and I’m rusty.  :)

Anyway, I disagree that it’s “about the relative strength of the shifts”. It’s not a “shift”, because the system has no equilibrium. If the AGI supply curve were stuck at C1, then over time, the AGI demand curve would settle to C2. Separately, if the AGI demand curve were stuck at C2, then over time, the AGI supply curve would settle to C3, which is to the right of C1. See what I mean? The two curves just race each other outwards, forever, until the surface of Earth is plastered with solar cells and factories etc. So, to figure out the current price, it matters which curve can shift faster in response to current conditions.

Well, if the economy has a set number of jobs, you'd expect a lot of human labor displaced, but if the economy can find other useful work for those people, they will do those other jobs

I think that, in this quote, you’re being the person on the left side of my silly diagram :) You’re assuming that the economy will produce new jobs faster than the factories will produce new chips and robots to fill those jobs. New jobs don’t appear instantaneously, right? …But new chips also don’t appear instantaneously. Which one is less instantaneous? You need to make an argument. That’s my point. :)

Problem #2: Now I have to go searching for a way to rate-limit the api calls sent by evalugator. Can't just slam GoodFire's poor little under-provisioned API with as many hits per minute as I want!

Error code: 429 - {'error': 'Rate limit exceeded: 100 requests per minute'}

Update: searched evalugator for 'backoff' and found a use of the backoff lib. Added this to my goodfire_provider implementation:

import backoff
...
def on_backoff(details):
    if int(details.get("tries", 0)) % 3 == 0:
        print(f"Backing off {details['wait']:0.1f} seconds after {details['tries']}. Reason: {details['exception']}")
...
@backoff.on_exception(
    wait_gen=backoff.expo,
    exception=(
        openai.RateLimitError,
        openai.APIError,
    ),
    max_value=60,
    factor=1.5,
    on_backoff=on_backoff,
)
def execute(model_id: str, request: GetTextResponse):
...

I get the sense that you were just trying to allude to the ideas that--

Even if you have some kind of "alignment", blindly going full speed ahead with AI is likely to lead to conflict between humans and/or various human value systems, possibly aided by powerful AI or conducted via powerful AI proxies, and said conflict could be seriously Not Good.

Claims that "democratic consensus" will satisfactorily or safely resolve such conflicts, or even resolve them at all, are, um, naively optimistic.

It might be worth it to head that off by unspecified, but potentially drastic means, involving preventing blindly going ahead with AI, at least for an undetermined amount of time.

If that's what you wanted to express, then OK, yeah.

Yes. That’s really my central claim. All the other discussions over values is not me saying "look, we’re going to resolve this problem of human values in one lesswrong post". It was to point to the depth of the issue (and, one important and I think overlooked point, that it is not just Mistake Theory that raw clarity/intelligence can solve, there is a fundamental aspect of Conflict Theory we won’t be able to casually brush aside) and that it is not idle philosophical wandering.

I'm sorry that came off as unduly pugnacious. I was actually reacting to what I saw as similarly emphatic language from you ("I can't believe some of you..."), and trying to forcefully make the point that the alternative wasn't a bed of roses.

Don’t be sorry, it served its illustration purpose.

We lesswronger are a tiny point in the space of existing human values. We are all WEIRD or very close to that. We share a lot of beliefs that, seen from the outside, even close outside like academia, seems insane. Relative to the modal human who is probably a farmer in rural India or China, we may as well be a bunch of indistinguishable aliens.

And yet we manage to find scissor statements pretty easily. The tails come apart scarily fast.

It just seemed awfully glib and honestly a little combative in itself.

I don’t see how glib and combative "this post is already too long" is ?

"obvious" probably is, yes. My only defense is I don’t have a strong personal style, I’m easily influenced, and read Zvi a lot, who has the same manner of overusing it. I probably should be mindful to not do it myself (I removed at least two on drafting this answer, so progress !).

Well, yes, but I did say "as least not while the 'humans' involved are recognizably like the humans we have now". I guess both the Superhappies and the Babyeaters are like humans in some ways, but not in the ways I had in mind.

No, I mean recognizable humans having an AGI in their hand can decide to go the Superhappies way. Or Babyeaters way. Or whatever unrecognizable-as-humans way. The choice was not even on the table before AGI, and that represent a fundamental change. Another fundamental change brought by AGI is the potential for an unprecedented concentration of power. Many leaders had the ambition to mold humanity to their taste ; none had the capacity to.

Some people definitely have a lot of their self-worth and sense of prestige tied up in their jobs, and in their jobs being needed. But many people don't. I don't think a retail clerk, a major part of whose job is to be available as a smiling punching bag for any customers who decide to be obnoxious, is going to feel too bad about getting the same or a better material lifestyle for just doing whatever they happen to feel like every day.

I think a lot of people have that. There’s a even meme for that "It ain’t much, but it’s honest work".

All in one, I don’t think either of us has much more evidence that a vague sense of things anyway ? I sure don’t have.

I remember hearing things close to "my agency is meaningful if and only if I have to take positive, considered action to ensure my survival, or at least a major chunk of my happiness".

I think that’s the general direction of the thing we’re trying to point, yes ?

A medieval farmer who screw up is going to starve. A medieval farmer who does exceptionally well will have a surplus he can use on stuff he enjoys/finds valuable.

A chess player who screw up is going to lose some ELO points (and some mix of shame/disappointment). A chess player who does exceptionally well will gain some ELO points (and some mix of pride/joy).

If you give me the choice of living the life of a medieval farmer or someone who has nothing in his life but playing chess, I will take the former. Yes, I know it’s a very, very hard life. Worse in a lot of ways (if you give me death as a third choice, I will admit that death starts to become enticing, if only because if you throw me in a medieval farmer life I’ll probably end up dead pretty fast anyway). The generator of that choice is what I (and apparently others) are trying to point with Meaningfulness/Agency.

I think a lot of things we enjoy and value can be described as "growing as a person".

Does "growing as a person" sounds like a terminal goal to you ? It doesn’t to me.

If it’s not, what is it instrumental to ?

For me it’s clear, it’s the same thing as the generator of the choice above. I grow so I can hope to act better when there’s real stakes. Remove real stakes, there’s no point in growing, and ultimately, I’m afraid there’s no point to anything.

Is "real stakes" easier to grasp than Agency/Meaningfulness ? Or have I just moved confusion around ?

I've also heard plenty of people talk about "meaningfulness" in ways that directly contradict your definition.

Well, the problem is that there is so much concepts, especially when you want to be precise, and so few words.

My above Agency/Meaningfulness explanation does not match perfectly with the one in my previous answer. It’s not that I’m inconsistent, it’s that I’m trying to describe the elephant from different sides (and yeah, sure, you can argue, the trunk of the elephant is not the same thing as the leg of the elephant).

That being said I don’t think they point to completely unrelated concepts. All of those definitions above "positive, considered actions..." ? "Broad Sweep of History" ? its collective version ? Yeah, I all recognize them as parts of the elephant. Even the altruistic one, even if I find that one a bit awkward and maybe misleading. You should not see them as competing and inconsistent definitions, they do point to the same thing, at least for me.

Try to focus more on the commonalities, less on the distinctions ? Try to outline the elephant from the trunk and legs ?

I would like to extend this slightly by switching perspective to the other side of the coin. The drop-in remote worker is not a problem of anthropomorphizing AI, so much as it is anthropomorphizing the need in the first place. Companies create roles with the expectation people will fill them, but that is the habit of the org, not the threshold of the need.

Adoption is being slowed down considerably by people asking for AI to be like a person, so we can ask that person to do some task. Most companies and people are not asking more directly for an AI to meet a need. Figuring out how to do that is a problem to solve by itself, and there hasn't been much call for it to date.

You can't guarantee whether it will stop acting that way in the future, which is what is predicted by deceptive alignment.

yes, that's true. But in fact if your AI is merely supposed to imitate a human it will be much easier to prevent deceptive alignment because you can find the minimal model that mimics a human, and that minimality excludes exotic behaviors.

This is essentially why machine learning works at all - you don't pick a random model that fits your training data well, you pick the smallest one.

I am interested in the GRT from an agent foundations point of view, not because I want to make better thermostats.

An agent with a goal needs to use the means available to it in whatever way will achieve that goal. That is practically the definition of a control system. So you do actually want to build better thermostats, even if you haven't realised it.

I'm sure that GRT is pretty useless for most practical applications of control theory!

I'm sure that GRT is pretty useless, period.

Reducing entropy is often a necessary (but not sufficient) condition for achieving goals. A thermostat can achieve an average temperature of 25C by ensuring that the room temperature comes from a uniform distribution over all temperatures between 75C and -25C. But a better thermostat will ensure that the temperature is distributed over a narrower (lower entropy) distribution around 25C .

A worse thermostat will achieve an equally low entropy distribution around 40C. Reaching the goal is what matters, not precisely hitting the wrong target.

With a specific deadline and a specific threat of a nuclear attack on the US.

In other words, what Putin has already been doing more and more, but with a specific deadline attached?