LessWrong 2.0 Reader

First some points of agreement:

• I like that you're focusing on neglected approaches. Not much on the technical side seems promising to me, so I like to see exploration.
  • Skimming through your suggestions, I think I'm most keen on human augmentation related approaches - hopefully the kind that focuses on higher quality decision-making and direction finding, rather than simply faster throughput.
• I think outreach to Republicans / conservatives, and working across political lines is important, and I'm glad that people are actively thinking about this.
• I do buy the [Trump's high variance is helpful here] argument. It's far from a principled analysis, but I can more easily imagine [Trump does correct thing] than [Harris does correct thing]. (mainly since I expect the bar on "correct thing" to be high so that it needs variance)
  • I'm certainly making no implicit "...but the Democrats would have been great..." claim below.

That said, various of the ideas you outline above seem to be founded on likely-to-be-false assumptions.
Insofar as you're aiming for a strategy that provides broadly correct information to policymakers, this seems undesirable - particularly where you may be setting up unrealistic expectations.
 

Highlights of the below:

1. Telling policymakers that we don't need to slow down seems negative.
   1. I don't think you've made any valid argument that not needing to slow down is likely. (of course it'd be convenient)
2. A negative [alignment-in-the-required-sense tax] seems implausible. (see below)
   1. (I don't think it even makes sense in the sense that "alignment tax" was originally meant^[1], but if "negative tax" gets conservatives listening, I'm all for it!)
3. I think it's great for people to consider convenient possibilities (e.g. those where economic incentives work for us) in some detail, even where they're highly unlikely. Whether they're actually 0.25% or 25% likely isn't too important here.
   1. Once we're talking about policy advocacy, their probability is important.

More details:

A conservative approach to AI alignment doesn’t require slowing progress, avoiding open sourcing etc. Alignment and innovation are mutually necessary, not mutually exclusive: if alignment R&D indeed makes systems more useful and capable, then investing in alignment is investing in US tech leadership.

Here and in the case for a negative alignment tax [LW · GW], I think you're:

1. Using a too-low-resolution picture of "alignment" and "alignment research".
   1. This makes it too easy to slip between ideas like:
      1. Some alignment research has property x
      2. All alignment research has property x
      3. A [sufficient for scalable alignment solution] subset of alignment research has property x
      4. A [sufficient for scalable alignment solution] subset of alignment research that we're likely to complete has property x
   2. An argument that requires (iv) but only justifies (i) doesn't accomplish much. (we need something like (iv) for alignment tax arguments)
2. Failing to distinguish between:
   1. Alignment := Behaves acceptably for now, as far as we can see.
   2. Alignment := [some mildly stronger version of 'alignment']
   3. Alignment := notkilleveryoneism

In particular, there'll naturally be some crossover between [set of research that's helpful for alignment] and [set of research that leads to innovation and capability advances] - but alone this says very little.

What we'd need is something like:

• Optimizing efficiently for innovation in a way that incorporates various alignment-flavored lines of research gets us sufficient notkilleveryoneism progress before any unrecoverable catastrophe with high probability.

It'd be lovely if something like this were true - it'd be great if we could leverage economic incentives to push towards sufficient-for-long-term-safety research progress. However, the above statement seems near-certainly false to me. I'd be (genuinely!) interested in a version of that statement you'd endorse at >5% probability.

The rest of that paragraph seems broadly reasonable, but I don't see how you get to "doesn't require slowing progress".

On "negative alignment taxes":

First, a point that relates to the 'alignment' disambiguation above.
In the case for a negative alignment tax [LW · GW], you offer the following quote as support for alignment/capability synergy:

...Behaving in an aligned fashion is just another capability... (Anthropic quote from Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback)

However, the capability is [ability to behave in an aligned fashion], and not [tendency to actually behave in an aligned fashion] (granted, Anthropic didn't word things precisely here). The latter is a propensity, not a capability.

What we need for scalable alignment is the propensity part: no-one sensible is suggesting that superintelligences wouldn't have the ability to behave in an aligned fashion. The [behavior-consistent-with-alignment]-capability synergy exists while a major challenge is for systems to be able to behave desirably.

Once capabilities are autonomous-x-risk-level, the major challenge will be to get them to actually exhibit robustly aligned behavior. At that point there'll be no reason to expect the synergy - and so no basis to expect a negative or low alignment tax where it matters.

On things like "Cooperative/prosocial AI systems [LW · GW]", I'd note that hits-based exploration is great - but please don't expect it to work (and that "if implemented into AI systems in the right ways" is almost all of the problem). 
 

On this basis, it seems to me that the conservative-friendly case you've presented doesn't stand up at all (to be clear, I'm not critiquing the broader claim that outreach and cooperation are desirable):

• We don't have a basis to expect negative (or even low) alignment tax.
  • (unclear so far that we'll achieve non-infinite alignment tax for autonomous x-risk relevant cases)
• It's highly likely that we do need to slow advancement, and will need serious regulation.

Given our lack of precise understanding of the risks, we'll likely have to choose between [overly restrictive regulation] and [dangerously lax regulation] - we don't have the understanding to draw the line in precisely the right place. (completely agree that for non-frontier systems, it's best to go with little regulation)

I'd prefer a strategy that includes [policymakers are made aware of hard truths] somewhere.
I don't think we're in a world where sufficient measures are convenient.

It's unsurprising that conservatives are receptive to quite a bit "when coupled with ideas around negative alignment taxes and increased economic competitiveness" - but this just seems like wishful thinking and poor expectation management to me.
 

Similarly, I don't see a compelling case for:

that is, where alignment techniques are discovered that render systems more capable by virtue of their alignment properties. It seems quite safe to bet that significant positive alignment taxes simply will not be tolerated by the incoming federal Republican-led government—the attractor state of more capable AI will simply be too strong. 

Of course this is true by default - in worlds where decision-makers continue not to appreciate the scale of the problem, they'll stick to their standard approaches. However, conditional on their understanding the situation, and understanding that at least so far we have not discovered techniques through which some alignment/capability synergy keeps us safe, this is much less obvious.

I have to imagine that there is some level of perceived x-risk that snaps politicians out of their default mode.
I'd bet on [Republicans tolerate significant positive alignment taxes] over [alignment research leads to a negative alignment tax on autonomous-x-risk-capable systems] at at least ten to one odds (though I'm not clear how to operationalize the latter).
Republicans are more flexible than reality :).

1. ^{^}

   As I understand the term, alignment tax compares [lowest cost for us to train a system with some capability level] against [lowest cost for us to train an aligned system with some capability level]. Systems in the second category are also in the first category, so zero tax is the lower bound.
   
   This seems a better definition, since it focuses on the outputs, and there's no need to handwave about what counts as an alignment-flavored training technique: it's just [...any system...] vs [...aligned system...].
   
   Separately, I'm not crazy about the term: it can suggests to new people that we know how to scalably align systems at all. Talking about "lowering the alignment tax" from infinity strikes me as an odd picture.

Yes, I agree that there is this difference in few examples I gave, but I don't agree that this difference is crucial.

Even if the agent puts max effort to keep its utility function stable over time, there is no guarantee it will not change. Future is unpredictable. There are unknown unknowns. And effect of this fact is both:

1. it is true that instrumental goals can mutate
2. it is true that terminal goal can mutate

It seems you agree with 1st. I don't see the reason you don't agree with 2nd.

It's predictably censored on CCP-sensitive topics.

(In a different chat.) After the second question, it typed two lines (something like "There have been several attempts to compare Winnie the Pooh to a public individual...") and then overwrote it with "Sorry...".

My biggest concern with intent alignment of AGI is that we might run into the issue of AGI being used for something like a totalitarian control over everyone who doesn't control AGI. It becomes a source of nearly unlimited power. The first company to create intent-aligned AGI (probably ASI at that point) can use it to stop all other attempts at building AGI. At that point, we'd have a handful of people wielding incredible power. It seems unlikely that they'd just decide to give it up. I think your "big if" is a really, really big if.

But other than that, your plan definitely seems workable. It avoids the problem of value drift, but unfortunately it incurs the cost dealing with power-hungry humans.

For the sake of clarity, let's discuss expected utility functions, which I mentioned above (or "pragmatism functions", say) from strategies to numbers, as opposed to utility functions from world-states to numbers, in order to make it clear that the actual utility function of an agent doesn't change.

That's another one of the reasons that I wasn't persuaded by your new example; in your new example, the agent believes that its future self will still be trying to create paperclips (same terminal goal) and will be better at that thanks to its greater knowledge (different instrumental goals although it doesn't know what), but in your old example, the agent believes that its future self will be trying to destroy paperclips (opposite terminal goal). There's a difference between having the rule-of-thumb "my current list of incidental goals might be incomplete, I should keep an eye out for things that are incidentally good" and having the rule-of-thumb "I shouldn't try to protect my terminal goal from changes". The whole point of those rules of thumb is to fulfill the terminal goal, but the second rule of thumb is actively harmful to that.

I do think that the first rule of thumb would be prudent for an agent to have, to one extent or another, to be clear.

I just think that - stepping back from the new example, and revisiting the old example, which seems much more clear-cut - the agent wouldn't tolerate a change in its utility function, because that's bad according to its current utility function. This doesn't apply to the new example because the pragmatism function is a different thing that the agent is trying to improve (and thus change).
(I find myself again emphasizing the difference between terminal and instrumental. I think it's important to keep in mind that difference.)

It could be really interesting how the employemnt looks before and after the camp. 

Okay, but you're not comparing like with like. Terminator 2 is an action movie, and I agree that action movies have gotten better since the 1960s. But in terms of sci-fi concepts introduced per second, I would suspect 2001 has more. Some movies from the 1990s that are more straight sci-fi would be Gattaca or Contact, but I don't think many people would consider these categorically better than 2001.

One thing I’d be bearish on is visibility into the latest methods being used for frontier AI methods, which would downstream reduce the relevance of alignment research except for the research within the manhattan-like project itself. This is already somewhat true of the big labs eg. methods used for o1 like models. However, there is still some visibility in the form of system cards and reports which hint at the methods. When the primary intention is racing ahead of China, I doubt there will be reports discussing methods used for frontier systems.

Interesting but I've just skim so will need to come back. With that caveat made, I seem to have had a couple of thought that keep recurring for me that seem compatible or complementary with your thoughts.

First, where do we define the margin between public and private. It strikes me that a fair amount of social strife does revolve around a tension here. We live in a dynamic world so thinking that the sphere of private actions will remain static seem unlikely but as the world changed (knowledge, applied knowledge driving technology change, movement of people resulting in cultural transmission and tensions...) will be forces resulting in a change in the line between public and private. 

While I'm not entirely sure it is the best framing, I do think of this in the form of externalities. Negative externalities are the more challenging form. What I think starts happening is that we live in t=0 and some set of private activities are producing very little negative impacts on others. But we find by t=10 some of the elements in that set of private activities are now producing a large enough total negative external effect that:

1. People are able to start seeing cause and effect
2. The costs to others are sufficient to over come the organizational and transactional costs of using social infrastructures to seek relief for the harms. Those can be formal in the sense of courts and government law/regulation requests. But also informal in various forms -- the people engaging in the activities loose reputation, don't get invited to the good parties any more, people don't want to talk with them or be seen with them any more, perhaps even more aggressive responses. 

At some point either most accept that a new definition of "private" exists and the old ways have changed or society reached the point that those who have not adapted will be treated as criminal and removed from society.

The other thing I've been thinking about is related. One hears the where's my flying car, it's the 21st Century already quip now and then. But I think a better one might be: It's the 21st Century, why am I still living under and 18th Century form of government? 

I think these relate some of your post in that a lot of the social conflict you point to is driven by the shifting margin between public and private sphere of action. As that margin shifts people use the government to address those new conflicts within the society. But few if any governments differ substantially from those that have existed for centuries.  I would characterize that vision of government, even when thinking of representative democracies, as that of an actor/agent. Government takes actions, just like the private members of society do. It should function, as you say, in a neutral way. Part of the failing there comes from government, being an actor/agent, then has its own interests, agendas and biases. 

That government as an active participant contrasts a bit with how I think most people think of markets. Markets don't really do anything. They are simply an environment in which active entities come and interact with each other. Markets don't set price or quality or even really type of item -- these are all unplanned outputs. The market itself is indifferent to all those, it's neutral in the sense you use that term.

Well, in the 21st Century might we not think that how governments are structured  might also shift? While I am far from sure that the shift would be correctly called divestiture or privatization (which seems most people think of when talking about fixing government -- or for some calling for increasing what its already doing) I do think the shift might be away from an acting entity and more into some type of passive environment that has some commonality with markets. In a very real sense governments are already a type of market setting but not a price/money exchange one (the representatives are not quite but out bids and offers on votes) but clearly these is an demand mediation and supply process going on. But currently the market-like aspect of government is about integrating voter/members of society demands and then the government makes a decision and takes the actions it wants. I would think some areas might be suitable for taking out the government being the actor and let the actions be decentralized among the people. Probably not individual action, I suspect some sub-agent presence will exist to reduce organizational/transaction costs but certainly the process would look more market-like and be a more neural setting. That might well then remove a lot of the divisiveness and conflict we see with the existing "old school" forms of government.

That is all probably a bit poorly written and expressed but it's a quick dump of a couple of not fully thought out ideas.

Riders of Justice:    imdb.com/title/tt11655202/

Recognizing patterns in a mainly random world, psycho-therapeutic hacking strategies. Can't say much more without risking spoilers.