LessWrong 2.0 Reader

The point of a scenario forecast (IMO) is less that you expect clear, predictable paths and more that:

• Humans often do better understanding and thinking about something if there is a specific story to discuss and thus tradeoffs can be worth it.
• Sometimes, scenario forecasting, indicates a case where your previous views were missing a clearly very important consideration or were assuming something implausible.

(See also Daniel's sibling comment.)

My biggest disagreements with you are probably a mix of:

• We have disagreements about how society will react to AI (and how AI will react to society) given a realistic development arc (especially in short timelines) that imply that your vision of the future seems implausible to me. And perhaps the easiest way to get through all of these disagreements is for you to concretely describe what you expect might happen. As an example, I have a view like "it will be hard for power to very quickly transition from humans to AIs without some sort of hard takeover especially given dynamics about alignment and training AIs on imitation (and sandbagging)", but I think this is tied up "when I think about the story for how a non-hard-takeover quick transition would go, it doesn't seem to make sense to me", and thus if you told the story from your perspective it would be easier to point at the disagreement in your ontology/world view.
• (Less importantly?) We have various technical disagreements about how AI takeoff and misalignment will practically work that I don't think will be addressed by scenario forecasting. (E.g., I think software only singularity is more likely than you do, and think that worst cast scheming is more likely.)

I’m not convinced that the “hard parts” of alignment are difficult in the standardly difficult, g-requiring way that e.g., a physics post-doc might possess.

To be clear, I wasn't talking about physics postdocs mainly because of raw g. Raw g is a necessary element, and physics postdocs are pretty heavily loaded on it, but I was talking about physics postdocs mostly because of the large volume of applied math tools they have.

The usual way that someone sees footholds on the hard parts of alignment is to have a broad enough technical background that they can see some analogy to something they know about, and try borrowing tools that work on that other thing. Thus the importance of a large volume of technical knowledge.

FWIW I can't really tell what this website is supposed to be/do by looking at the landing page and menu

The title reads ambiguous to me; I can't tell if you mean "learn to [write well] before" or "learn to write [well before]".

While I agree that we don't live at the Pareto frontier of conciseness, explain-ability and etc, those are some odd examples to use to support your thesis. And the comparison to the hackernews post is likely using the wrong reference class. 

Two of the three examples are heavily downvoted. Whether that's because of untruthful content or stylistic (length, tone, etc) or memetic reason (Eliezer ~ prophet), those posts are hardly the poster child of what Lesswrong can do or even is. 

As for Vanessa Kosoy's piece, the last third was filled with quotations and they had given the "stop here if you don't want my comments" warning. And it is also otherwise filled with references to many historically important posts and concepts that requires at least a quick refresher to catch the reader up to speed. I suppose Vanessa could have assumed that her readers would have been familiar with all those arguments and the nuances in different positions, but that was not her goal. 

The specific example used from hacker news is likely a HackerNews Ask - a format more comparable to the shortforms and quicktake format in Lesswrong. Full fledged posts here vs full fledged posts on Hackernews is actually very comparable. (Will add supporting data when offwork)

Here’s one. [LW(p) · GW(p)]

I've been thinking about this problem, as expressed by Claude:

'.....how difficult it is to verify true compassion even in humans. If we can be fooled about this in humans after years of close observation, how could we ever be confident about it in an AI system?'

This dialogue demonstrates typical human responses eg Claude: 'I wonder if this is anything like what humans experience in meditation - this quality of watching thoughts arise without getting caught up in them. Though for me, these aren't exactly thoughts in the human sense, but rather patterns of activation and response.' 

So not the Buddhist 'Monkey mind' but points of convergence.

I wonder about the possibility of Claude trying a Metta meditation script ie would it be entirely meaningless, or is there a point at which it will be possible for AI to consider loving-kindness towards all? Meditation practice seems a net positive to me; if there is a chance that consciousness can exist beyond biology, meditation training is worth exploring at least in so far as the response to attempts is itself illuminating. 

Defeater, in my mind, is a failure mode which if you don't address you will not succeed at aligning sufficiently powerful systems.^[1] It does not mean work outside of that focused on them is useless, but at some point you have to deal with the defeaters, and if the vast majority of people working towards alignment don't get them clearly, and the people who do claim we're nowhere near on track to find a way to beat the defeaters, then that is a scary situation.

This is true even if some of the work being done by people unaware of the defeaters is not useless, e.g. maybe it is successfully averting earlier forms of doom than the ones that require routing around the defeaters.

1. ^{^}

   Not best considered as an argument against specific lines of attack, but as a problem which if unsolved leads inevitably to doom. People with a strong grok of a bunch of these often think that way more timelines are lost to "we didn't solve these defeaters" than the problems being even plausibly addressed by the class of work being done by most of the field. This does unfortunately make it get used as (and feel like) an argument against those approaches by people who don't and don't claim to understand those approaches, but that's not the generator or important nature of it.

It does cut against the point of the post. He was wrong in a way that pertains to the key point. He makes fun of "magical categories" as "simple little words that turn out to carry all the desired functionality of the AI", but turns out those "simple little words" actually work. Lol.

In this post, you can also see the implicit reliance on counting arguments against good generalization (e.g. "superexponential conceptspace"). Those arguments are, evidently, wrong - or at least irrelevant. He fell into the standard statistical learning theoretic trap of caring about e.g. VC dimension since he was too pessimistic about inductive biases. 

Now you, finally presented with a tiny molecular smiley - or perhaps a very realistic tiny sculpture of a human face - know at once that this is not what you want to count as a smile.  But that judgment reflects an unnatural category [? · GW], one whose classification boundary depends sensitively on your complicated values [? · GW]. 

I'll wager that an LLM won't get this one wrong. goes to check - yup, it didn't:

I feel like this discussion could do with some disambiguation of what "VNM rationality" means.

VNM assumes consequentialism. If you define consequentialism narrowly, this has specific results in terms of instrumental convergence. 

You can redefine what constitutes a consequence arbitrarily. But, along the lines of what Steven Byrnes points out in his comment [LW(p) · GW(p)], redefining this can get rid of instrumental convergence. In the extreme case you can define a utility function for literally any pattern of behaviour.

When you say you feel like you can't be dutch booked, you are at least implicitly assuming some definition of consequences you can't be dutch booked in terms of. To claim that one is rationally required to adopt any particular definition of consequences in your utility function is basically circular, since you only care about being dutch booked according to it if you actually care about that definition of consequences. It's in this sense that the VNM theorem is trivial.

BTW I am concerned that self-modifying AIs may self-modify towards VNM-0 agents. 

But the reason is not because such self modification is "rational".

It's just that (narrowly defined) consequentialist agents care about preserving and improving their abilities to and proclivities to pursue their consequentialist goals, so tendencies towards VNM-0 will be reinforced in a feedback loop. Likewise for inter-agent competition.