LessWrong 2.0 Reader

Not sure I understand what you mean by that. The Universe seems to follow relatively simple deterministic laws. That doesn't mean you can use quantum field theory to predict the weather. But chaotic systems can be modeled as statistical ensembles. Temperature is a meaningful measurement even if we can't calculate the motion of all the individual gas molecules.

If you're referring to human irrationality in particular, we can study cognitive bias [? · GW], which is how human reasoning diverges from that of idealized agents in certain systematic ways. This is a topic of interest at both the individual level of psychology, and at the level of statistical ensembles in economics.

In terms of developing better misalignment risk countermeasures, I think the most important questions are probably:

• How to evaluate whether models should be trusted or untrusted: currently I don't have a good answer and this is bottlenecking the efforts to write concrete control proposals.
• How AI control should interact with AI security tools inside labs.

More generally:

• How can we get more evidence on whether scheming is plausible?
• How scary is underelicitation? How much should the results about password-locked models [LW · GW] or arguments about being able to generate small numbers of high-quality labels or demonstrations [AF · GW] affect this?

Curated. I think this model is pretty useful and well-compressed, and I'm glad to be able to concisely link to it.

The policy implications are still much open to debate, for here on LessWrong and for other ecosystems in the world.

I have observed a transition. 12 years ago, the left-right split was based on many loosely correlated factors and strategic/inertial effects, creating bizarre situations like near perfect correlation between opinions on Gay Marriage and privatization of social security. I think at that time you could reason much better if you could recognize that the separation between left and right was not natural. I at least have a ton of cached arguments from this era because it became such a familiar dynamic. 

Nowadays, I don't think this old schema really applies, especially among the actual elected officers and party leadership. The effective left right split is mono-factor: you are right exactly in proportion to your personal loyalty to one Donald J. Trump, resulting in bizarre situations like Dick Cheney being classified as "Left."  

 

That no one rebuilt old OkCupid updates me a lot about how much the startup world actually makes the world better

The prevailing ideology of San Francisco, Silicon Valley, and the broader tech world, is that startups are an engine (maybe even the engine) that drives progress towards a future that's better than the past, by creating new products that add value to people's lives.

I now think this is true in a limited way. Software is eating the world, and lots of bureaucracy is being replaced by automation which is generally cheaper, faster, and a better UX. But I now think that this narrative is largely propaganda.

It's been 8 years since Match bought and ruined OkCupid and no one, in the whole tech ecosystem, stepped up to make a dating app even as good as old OkC is a huge black mark against the whole SV ideology of technology changing the world for the better.

Finding a partner is such a huge, real, pain point for millions of people. The existing solutions are so bad and extractive. A good solution has already been demonstrated. And yet not a single competent founder wanted to solve that problem for planet earth, instead of doing something else, that (arguably) would have been more profitable. At minimum, someone could have forgone venture funding and built this as a cashflow business.

It's true that this is a market that depends on economies of scale, because the quality of your product is proportional to the size of your matching pool. But I don't buy that this is insurmountable. Just like with any startup, you start by serving a niche market really well, and then expand outward from there. (The first niche I would try for is by building an amazing match-making experience for female grad students at a particular top university. If you create a great experience for the women, the men will come, and I'd rather build an initial product for relatively smart customers. But there are dozens of niches one could try for.)

But it seems like no one tried to recreate OkC, much less creating something better, until the manifold team built manifold.love (currently in maintenance mode)? Not that no one succeeded. To my knowledge, no else one even tried. Possibly Luna counts, but I've heard through the grapevine that they spent substantial effort running giant parties, compared to actually developing and launching their product—from which I infer that they were not very serious. I've been looking for good dating apps. I think if a serious founder was trying seriously, I would have heard about it. 

Thousands of funders a year, and no one?!

That's such a massive failure, for almost a decade, that it suggests to me that the SV ideology of building things that make people's lives better is broadly propaganda. The best founders might be relentlessly resourceful, but a tiny fraction of them seem to be motivated by creating value for the world, or this low hanging fruit wouldn't have been left hanging for so long.

This is of course in addition to the long list of big tech companies who exploit their network-effect monopoly power to extract value from their users (often creating negative societal externalities in the process), more than creating value for them. But it's a weaker update that there are some tech companies that do ethically dubious stuff, compared to the stronger update that there was no startup that took on this obvious, underserved, human problem.

My guess is that the tech world is a silo of competence (because competence is financially rewarded), but operates from an ideology with major distortions / blindspots, that are disconnected from commonsense reasoning about what's Good. eg following profit incentives, and excitement about doing big things (independent from whether those good things have humane or inhumane impacts) off a cliff.

To me memetic normally reads something like "has a high propensity to become a meme" or "is meme-like" I had no trouble interpreting the post from this basis.

I push back against trying to hew closely to usages from the field of genetics. Fundamentally I feel like that is not what talking about memes is for; it was an analogy from the start, not meant for the same level of rigor. Further, memes and how meme-like things are is much more widely talked about than genetics, so insofar as we privilege usage considerations I claim switching to one matching genetics would require more inferential work from readers overall because the population of readers conversant with genetics is smaller.

I also feel like the value of speaking in terms of memes in the post is that the replication crises is largely the fault of non-rigorous treatment; that is to say in many fields the statistical analysis parts really were/are more of a meme inside the field rather than a rigorous practice. People just read other people's published papers analysis sections, and write something shaped like that, replicability be damned.

I don't think this makes any sense. How are you hoping to get the model to attack except by telling it that it's in a control evaluation and you want it to attack? It seems that you are definitely going to have to handle the sandbagging.

Hi notfnofn, thanks again for the well considered comment, and for responding to my edited response. I think you've made good points which have revealed clarifications I could have made within the post.

Okay Trump is president now. Hoping that things go well regardless.

Me too. And we'll see if the right-wing and online media's concern that Harris is an equal threat to democracy over the next couple of months. Because if she is an equal threat we shouldn't expect to see a peaceful transfer of power, like when Trump lost. Although, she has already graciously conceded as would be expected of any political candidate except Trump (who has continued to lie about the result of the 2020 election and require his followers and compatriots to do the same) due to the fact that he is held to a different standard. Obviously no one seriously expects Harris to lead an insurrection on the capitol, but they have been convinced that both-sides are equally dangerous, giving a permission structure to vote for Trump.

It's not necessarily that it was the worst issue, but the easiest target.

First of all, I am part of the majority that believe that trans-women shouldn't be competing the women's category in sport. It's dangerous, and undermines the integrity of the category due to the natural physical advantages of being born male, particularly on the extremes.

But my point is, as you say "it's not necessarily the worst issue" whereas the promise to "root out" the "enemy within" is the literally the worst issue. The radical left want to fight for the rights of trans-people in all areas, and unfortunately, I believe, have over-stepped in terms of sport—an entirely optional recreational activity of little to no consequence in my opinion. This is an issue that is adjudicated largely independently of government by international sports' bodies, and I hope that over time a fair and consistent ruling will prevail.

Rooting out the enemy within, on the other hand, is not even considered radical on the right, it's said out loud by a mainstream candidate with popular support. This is how far the centre has shifted.

I'm a bit concerned that you referred to cancel culture as "accountability culture"

I think this is a fair use of both-sidesism, if I'm going to use the loaded term 'cancel culture', I'm going to qualify that this is opposed by others who see this as 'accountability culture'. I'm a believer in the free market of ideas, and my support for this principle doesn't stop when a group of less powerful people collectively use their ideas to combat powerful individuals, I also think companies should be able to act so as to protect themselves from public backlash—I largely believe in free markets in general. Where there are instances of top-down cancelling, which as you mention happens on both sides, I'm opposed to this, and would happily call this cancel culture without qualification. But in my experience that's a small proportion of what people call "cancel culture".

Do you not see this as a false equivalence?—Yes...

Great.

Are you comparing the opinions of US politicians on the left with US politicians on the right?

I'm comparing activists on the left with activists on the right. Both the Democratic and Republican parties profess strong pro-Israel support.

How seriously have you investigated the claim that "Harris's plan is based on what many top economists think is best" and not "Economists find Harris' plan overall better than Trump's, despite its many weaknesses"?  Have you controlled for the likelihood that they have other reasons to prefer Kamala to Trump?

The first I'd heard of this was in the debate, as a claim of Harris' that Goldman Sachs and the Wharton School supported her plan, and that 16 Nobel laureates had said that Trump's plan would invite a recession and increase the deficit. This demonstrated her respect for those experts. Trump wasn't able to make any similar claims. Since then I have tried to understand more about tariffs, looking to the Wall St Journal and their explanation of Tariffs, Trump's own interview with John Micklethwait, where he claimed the room full of economists didn't understand tariffs, and this interview with The Economist Editor in Chief Zanny Minton Beddoes where she underscores the strength of Harris's plan relative to Trump's.

These are all respected, relatively right-leaning sources who all agree with Harris, and who's opinions are respected by Harris, as opposed to Trump who has shown disdain for the opinion of the majority of these experts, in favour of his own expertise, borne out of his experience going bankrupt 6 times. I expect that when developing their plans, this same respect for expertise was also at play. So, I think I've investigated this claim seriously enough to have a fair opinion on it.

I'd like to thank you again for this response. I believe you've raised important clarifications that I will consider making in the text itself. As you might know, this cross-posted from my blog, and the blog is actually a series of webpages that I edit continually comprising a growing philosophical framework, and I will likely attempt to make it more ever-green by relying less on a current event. Posting here helps guide my editing process by getting critical feedback from smart people like yourself, so I appreciate your time and efforts.

My prior intuitive guess would be that H1 seems quite a decent chunk more likely than H2 or H3.

Actually I changed my mind.

Why I thought this before: H1 seems like a potential runaway-process and is clearly about individual selection which has stronger effects than group selection (and it was mentioned in HPMoR).

Why I don't think this anymore:

• It would also be incredibly huge coincidence if intelligence mostly evolved because of social dynamics but happened to be useful for all sorts of other survival techniques hunters and gatherers use. See e.g. Scott Alexander's Book review of "The Secret of our success" [LW · GW].
• If there was only individual benefits for intelligence but it was not very useful otherwise then over long timelines group selection would actually select against smarter humans because their neurons would use up more metabolic energy.

However, there's a possibly very big piece of evidence for H3: Humans are both the smartest land animals and have the best interface for using tools, and that would seem like a suspicious coincidence.

I think this is not a coincidence but rather that tool use let humans fall into an attractor basin where payoffs of intelligence were more significant.

I'd say 1 important question is whether the AI control strategy works out as they hope.

I agree with Bogdan that making adequate safety cases for automated safety research is probably one of the most important technical problems to answer (since conditional on the automating AI safety direction working out, then it could eclipse basically all safety research done prior to the automation, and this might hold even if LWers really had basically perfect epistemics given what's possible for humans, and picked closer to optimal directions, since labor is a huge bottleneck, and allows for much tighter feedback loops of progress, for the reasons Tamay Besiroglu identified):

https://x.com/tamaybes/status/1851743632161935824

https://x.com/tamaybes/status/1848457491736133744