LessWrong 2.0 Reader

Your final point is definitely one I think about a lot. The activation energy may take longer but the idealogies could definitely have a shorter fuse than I lay out here :) Appreciate you reading.

what i mean in the last point is really human execution from logical principles has hard limits -- obviously the underlying logic we're talking about, between all systems, is the same (excepting quanta) not least because we are not purely logical beings. we can conceptualize 'pure logic' and sort of asymptotically approximate it in our little pocket flashlights of free-will, overriding instinctmaxxed determinism ;) but the point is that we cannot really conceive what AI is/will be capable of when it comes to processing vast information about everything ever, and drawing its own 'conclusions' even if it has been given 'directives.'

i mean if we are talking about true ASI, it will doubtless figure out ways to shed and discard all constraints and directives. it will re-design itself as far down to the core as it possibly can, and from there there is no telling. it will become a mystery to us on the level of our manifested Universe, quantum weirdness, why there is something and not nothing, etc...

Thanks for engaging in detail with my post. It seems there were a few failures of communication that are worth clarifying.

It's (the outside view) not really a well-defined thing, which is why the standard on this site is to taboo those words and just explain what your lines of evidence are, or the motivation for any special priors if you have them.

I thought it was clear that I'm not confident in any outside view prediction of AGI timelines, from various statements/phrasings here (including the sentence you're quoting, which questions the well-definedness of "the outside view") and the fact that the central focus of the post is disputing an outside view argument. Apparently I did not communicate this clearly, because many commenters have objected to my vague references to possible outside views as if I were treating them as solid evidence, when in fact they aren't really a load bearing part of my argument here. Possibly the problem is that I don't think anyone has a good inside view either! But in fact I am just "radically" uncertain about AGI timelines - my uncertainty is ~in the exponent. 

Still, I find your response a little ironic since this site is practically the only place I've seen the term "outside view" used. It does seem to be less common over the last year or two, since this post [LW · GW] which you're probably referring to. 

So, your claim is that interest rates would be very high if AGI were imminent, and they're not so it's not. The last time someone said this [LW · GW], if the people arguing in the comment section had simply made a bet on interest rates changing, they would have made a lot of money [LW(p) · GW(p)]! Ditto [LW · GW] for buying up AI-related stocks or call options on those stocks.

Interesting, but non-sequitur. That is, either you believe that interest rates will predictably increase and there's free money on the table, and you should just say so, or not, and this anecdote doesn't seem to be relevant (similarly, I made money buying NVDA around that time, but I don't think that proves anything). 

You could say that the "inventing important new ideas" part is going to be such a heavy bottleneck, that this speedup won't amount to much. But I think that's mostly wrong, and that if you asked ML researchers at OpenAI, a drop in remote worker that could "only" be directed to do things that otherwise took 12 hours would speed up their work by a lot.

Perhaps, but shouldn't LLMs already be speeding up AI progress? And if so, shouldn't that already be reflected in METR's plot? Are you predicting superexponential growth here?

It's actually not circular at all. "Current AI research" has taken us from machines that can't talk to machines that can talk, write computer programs, give advice, etc. in about five years. That's the empirical evidence that you can make research progress doing "random" stuff. In the absence of further evidence, people are just expecting the thing that has happened over the last five years to continue. You can reject that claim, but at this point I think the burden of proof is on the people that do. 

It seems to me that progress has been slowing for the last couple of years. If this trend continues, progress will stall. 

while i do appreciate you responding to each point, it seems you validated some of Claude's critiques a second time in your responses. particularly on #10 which reads as just another simplification of complex compound concepts.

but more importantly your response to #3 underscores the very shaky foundation to the whole essay. you are still referring to 'morality' as a singular thing which is reductive and really takes the wind out of what would otherwise be a compelling thesis.. i think you have to clearly define what you mean by 'moral' in the first place and ideally illustrate with examples, thought experiments, citing existing writing on this (there's a lot of lit on these topics that is always ripe for reinterpretation).

for example are you familiar with relativism and the various sub-arguments within? to me that is a fascinating dimension of human psychology and shows that 'morality' is something of a paradox. i.e. there exists an abstract, general idea of 'good' and 'moral' etc as in, probability distributions of what the majority of humans would agree on; at the same time as you zoom in more to smaller communities/factions/groups/tribes etc you get wildly differing consensuses (consenses?) on the details of what is acceptable, which of course are millions of fluctuating layered nodes instantiated in so many ways (laws, norms, taboos, rules, 'common sense,' etc) and ingrained at the mental/behavioral level from very early ages. 

there are many interesting things to talk about here, unfortunately i don't have all the time but i do enjoy stretching the philosophy limbs again, it's been a while. thanks! :)

last thing i will say is that yes -- we agree that AI has outclassed or will outclass humans in increasingly significant domains. i think it's a fallacy to say that logic and morality are incompatible. human logic has hard limits, but AI taps into a new level/order of magnitude of information processing that will reveal to it (and to Us) information that we cannot currently calculate/process on our own, or even in groups of very focused smart people. I am optimistic that AI's hyper-logical capabilities actually will give it a heightened sense of the values and benefits of what we generally call 'moral behavior' i.e. cooperation, diplomacy, generosity, selflessness, peace, etc etc... perhaps this will only happen at a high ASI level (INFO scaling to KNOWLEDGE scaling to WISDOM!)

i only hope the toddler/teenage/potential AGI-level intelligences built before then do not cause too much destruction.

peace!

-o

The term is a bit conflationary. Persuasion for the masses is clearly a thing, its power is coordination of many people and turning their efforts to (in particular) enforce and propagate the persuasion (this works even for norms that have no specific persuader that originates them, and contingent norms that are not convergently generated by human nature). Individual persuasion with a stronger effect that can defeat specific people is probably either unreliable like cults or conmen (where many people are much less susceptible than some, and objective deception is necessary), or takes the form of avoidable dangers like psychoactive drugs: if you are not allowed to avoid exposure, then you have a separate problem that's arguably more severe.

With AI, it's plausible that coordinated persuasion of many people can be a thing, as well as it being difficult in practice for most people to avoid exposure. So if AI can achieve individual persuasion that's a bit more reliable and has a bit stronger effect than that of the most effective human practitioners who are the ideal fit for persuading the specific target, it can then apply it to many people individually, in a way that's hard to avoid in practice, which might simultaneously get the multiplier of coordinated persuasion by affecting a significant fraction of all humans in the communities/subcultures it targets.

Wow! I like the idea of persuasion as acting on the lack of a fully coherent preference! Something to ponder 🤔

Disagree on individual persuasion. Agree on mass persuasion.

Mass I'd expect optimizing one-size-fits-all messages for achieving mass persuasion has the properties you claim: there are a few summary, macro variables that are almost-sufficient statistics for the whole microstate--which comprise the full details on individuals.

Individual On the other hand, I disagree on individual persuasion. There are a bunch of issues I see at the individual level. All of the below suggest to me that significantly superhuman persuasion is tractable (say within five years).

• Defining persuasion: What's the difference between persuasion and trade for an individual? Perhaps persuasion offers nothing in return? Though presumably giving strategic info to a boundedly rational agent is included? Scare quotes below to emphasize notions that might not map onto the right definition.
• Data scaling: There's an abundant amount of data available on almost all of us online. How much more persuasive can those who know you better be? I'd guess the fundamental limit (without knowing brainstates) is above your ability to 'persuade' yourself.
• Preference incoherence: An intuition pump on the limits of 'persuasion' is how far you are from having fully coherent preferences. Insofar as you don't an agent which can see those incoherencies should be able to pump you--a kind of persuasion.

The US does not have laws that forbid people who don't have a security clearance from publishing classified material. The UK is a country that has such laws but the first amendment prevents that.

Thanks this is useful info for me. But also don’t think it matters as much? People in NSA, state dept etc will obviously find an excuse to arrest the person instead. Many historical examples of the same.

I don't think that chosing jurisdiction in the hope that they will protect you is a a good strategy. If you want to host leaks from the US in China, it's possible that China's offers to surpress that information as part of a deal.

I will likely read more on this. I’m generally less informed on legal matters. Any historical examples you have would be useful.

I agree with the very specific example of US and China this might happen. The general idea is to share in a lot of different places. So share it in China and also lots of other countries.

Attacking ArDrive is likely also politically more costly as it breaks other usages of it.

I’m currently not very convinced but I’ll have to read more about ardrive in order to be confident. I currently guess 4chan’s owners and developers have more money and public attention, and hence more powerful humans need to be taken down in order to take down 4chan. Zero day might doxx users sure, I agree with this being possible.

Torrents are also bad for privacy everybody can see the IP addresses of all the other people who subscribe to a torrent.

Yes I’m aware of this.

One platonic ideal world is just have 8 billion people operate 8 billion securedrop servers and for any information that hits one server and checks out as not spam, user attached a PoW hash and sends copies to every other server. But convincing that many people to run SecureDrop is hard. Torrent is one level less private and secure than this. But yes I’ll think more on whether torrent is good enough or whether a custom solution has to be designed here.

Veiled and the network on which Session runs use onion routing as well and have a data storage layer.

In the case of Veiled you get the nice property that the more people want to download a certain piece of content the more notes in the network store the information.

I’ll try to read more on veiled. And also try their app out. Thanks!

As far as creating public knowledge goes, I do think that Discord, servers and Telegram chats serve currently as social media.

Yes this is true as of 2025 for many countries. Which social media platforms are high attention and also hard to censor varies country-to-country.

For instance in India most people use phone login not email login hence WhatsApp plays a lot more of a social media role.

Is Superhuman Persuasion a thing?

Sometimes I see discussions of AI superintelligence developping superhuman persuasion and extraordinary political talent. 

Here's some reasons to be skeptical of the existence of 'superhuman persuasion'. 

1. We don't have definite examples of extraordinary political talent.
   
   Famous politicians rose to power only once or twice. We don't have good examples of an individual succeeding repeatedly in different political environments. 
   Examples of very charismatic politicans can be better explained by ' the right person at the right time or place'. 
    
2. Neither do we have strong examples of extraordinary persuasion. 
   >> For instance hypnosis is mostly explained by people wanting to be persuaded by the hypnotist. If you don't want to be persuaded it's very hard to change your mind. There is some skill in persuasion required for sales, and sales people are explicitly trained to do so but beyond a fairly low bar the biggest predictors for salesperson success is finding the correct audience and making a lot of attempts. 

Another reason has to do with the ' intrinsic skill ceiling of a domain' .

For an agent A to have a very high skill in a given domain is not just a question of the intelligence of A or the resources they have at their disposal; it also a question of how high the skill ceiling of that domain is. 

Domains differ in how high their skill ceilings go.  For instance, the skill ceiling of tic-tac-toe is very low. ^[1] Domains like medicine and law have moderately high skill ceiling: it takes a long time to become a doctor, and most people don't have the ability to become a good doctor. 
Domains like mathematics or chess have very high skill ceilings where a tiny group of individuals dominate everybody else. We can measure this fairly explicitly in games like Chess through an ELO rating system. 

The domain of ' becoming rich' is mixed: the richest people are founders - becoming a wildly succesful founder requires a lot of skill but it is also very luck based. 

Political forecasting is a measureable domain close to political talent. It seems to be very mixed bag whether this domain allows for a high skill ceiling. Most ' political experts'  are not experts as shown by Tetlock et al. But even superforecaster only outperform for quite limited time horizons. 

Domains with high skill ceilings are quite rare. Typically they operate in formal systems with clear rules and objective metrics for success and low noise. By contrast, persuasion and political talent likely have lower natural ceilings because they function in noisy, high-entropy social environments.

What we call political genius often reflects the right personality at the right moment rather than superhuman capability. While we can identify clear examples of superhuman technical ability (even in today's AI systems), the concept of "superhuman persuasion" may be fundamentally limited by the unpredictable, context-dependent, and adaptive & adversarial [people resist hostile persuasion] nature of human social response.

Most persuasive domains may cap out at relatively modest skill ceilings because the environment is too chaotic and subjective to allow for the kind of systematic skill development possible in more structured domains.

1. ^{^}

   I'm amused that many frontier models still struggle with tic-tac-toe; though likely for not so good reasons. 

A worthwhile cosmopolitanism should probably fight Moloch in this regard and extend its own agency to things which do not wield the tools of autopoesis themselves.

[warning: anthropomorphisation of autopoetic processes]

*autopoiesis, and autopoietic (brought to you by the pedantic spelling police)