LessWrong 2.0 Reader

I probably agree with you guys more than you realize. You guys might be interested in this DM I sent to nostalgebraist:

Hey! Appreciated your post about AI 2027: https://nostalgebraist.tumblr.com/post/779943693916553216/so-about-this-new-ai-2027-report-i-have-not

Overall, I agree that we were not prioritizing addressing the demographic of skeptics who have detailed reasons for their beliefs. I very much sympathize with disagreeing the framework that others are using to approach takeoff forecasting rather than just parameter settings, I feel similarly to some extent about others' work (in particular Epoch's GATE).

However, I disagree that our model assumes a software-driven intelligence explosion. A substantial percentage of our simulations don't contain such an explosion! You can see that for example the 90th percentile for ASI is >2100. You can totally input your best guesses for the parameters in our model and end up with <50% on a software-driven explosion.

I also think that our scenario and supplements have convinced some skeptics who didn't have as sophisticated as reasons as e.g. you and Epoch for their disagreements going in. But of course you probably think this is a bad thing so :shrug:

And I’m also worried – as always with this stuff – that there are some people who will look at all those pages and pages of fancy numbers, and think “wow! this sounds crazy but I can’t argue with Serious Expert Research™,” and end up getting convinced even though the document isn’t really trying to convince them in the first place.

Also very much sympathize with this! I've had similar concerns about other approaches. I aimed to try to be clear about our levels of uncertainty and how scrappy the model is but perhaps I could have added further caveats? Curious what you would have recommended here.

Finally: I'd be quite interested in your object-level disagreements with our takeoff forecasts. I've appreciated your comments on timelines. I'd also be interested in your actual forecasts on timelines/takeoff. Like e.g. your 10/50/90th percentiles, as we gave in our supplements.

Yeah, at this point the marginal value add of forecasting/epistemics is in validating/invalidating fundamental assumptions like the software intelligence explosion idea, or the possibility of industrial/chip factory being massively scaled up by AIs, or Moore's law not ending, rather than on the parameter ranges, because the assumptions overdetermine the conclusion.

Comment down below:

https://forum.effectivealtruism.org/posts/rv4SJ68pkCQ9BxzpA/?commentId=EgqgffC4F5yZQreCp [EA · GW]

I don't know what legible/transferable evidence would be. I've audited a lot of courses at a lot of different universities. Anecdote, sorry.

Despite (2), it's important to try anyways

FWIW this is the step I disagree with, if I understand what you mean by "try". See this post [LW · GW].

Forecasts which include intuitive estimations are commonplace and often useful (see e.g. intelligence analysis, Superforecasting, prediction markets, etc.).

In this context, we're trying to forecast radically unprecedented events, occurring on long subjective time horizons, where we have little reason to expect these intuitive estimates to be honed by empirical feedback. Peer disagreement is also unusually persistent in this domain. So it's not at all obvious to me that, based on superforecasting track records, we can trust that our intuitions pin down these parameters to a sufficient degree of precision.^[1] More on this here [LW · GW] (this is not a comprehensive argument for my view, tbc; hoping to post something spelling this out more soon-ish!).

1. ^{^}

   As the linked post explains, "high precision" here does not mean "the credible interval for the parameter is narrow". It means that your central/point estimate of the parameter is pinned down to a narrow range, even if you have lots of uncertainty.

Suprisingly 6 different users downvoted a straightforward comment without explanation… so I imagine that also indicates a bot problem which has to be factored in, (or at least users replicating bot behavior) 

It's straightforward to disprove: they should be able to argue for their views in a way that stands up to scrutiny.

I'd like to see more intellectual scenes that seriously think about AGI and its implications. There are surely holes in our existing frameworks, and it can be hard for people operating within them to spot. Creating new spaces with different sets of shared assumptions seems like it could help.

Absolutely not, no, we need much better discovery mechanisms for niche ideas that only isolated people talk about, so that the correct ideas can be formed.

Most of your examples seem more like "prerequisites" or basic skills that you build on. But scaffolding is a thing you build up to get something else done, then get rid of afterwards. So, a scaffolding skill would be a skill that enables you to learn how to do something you actually want to learn, but once you have learned how to do that thing, you no longer need the scaffolding skill.

Algebraic notation can still be useful to a chess player. Knowing basics like how to properly cut things is integral to cooking. Debugging is an essential skill for programming. Etc.

A couple better examples of scaffolding skills:

• In calculus, learning to calculate a derivative using limits. Once you have the concept of derivatives down, you wouldn't go through that exercise, you would use the the various formulas (or a math program) to actually calculate them.
• When trying to get a business group to adopt Agile methodology, using strict Agile Scrum, which gives a bunch of prescriptive processes, and demonstrate how to "do Agile". But, teams that have internalized the Agile philosophy tend to ditch many of those processes (or at least strict adherence to them) as they move toward more efficient approaches, tailored to their situation.

Assumptions that highly constrain the model should be first and foremost rather than absent from publicly facing write-ups and only in appendices.

Strongly agree — cf. nostalgebraist's posts making this point on the bio anchors and AI 2027 models. I have the sense this is a pretty fundamental epistemic crux between camps of people making predictions (or suspending judgment!) about AI takeoff.

I'm not sure I buy that they will be more cautious in the context of an "arms race" with a foreign power. The Soviet Union took a lot of risks their bioweapons program during the cold war.

My impression is the CCP's number one objective is preserving their own power over China. If they think creating ASI will help them with that, I fully expect them to pursue it (and in fact to make it their number one objective)