LessWrong 2.0 Reader

I hadn't downvoted this post, but I am not sure why OP is surprised given the first four paragraphs, rather than explaining what the post is about, instead celebrate tree murder and insult their (imagined) audience:

so that no references are needed but those any LW-rationalist is expected to have committed to memory by the time of their first Lighthaven cuddle puddle

I think austerity has a weird relationship with counterfactuals?

It's not clear why anyone would want to claim a self-diagnosis of that, since little about it is 'egosyntonic', as the psychiatrists say.

Since a friend mentioned I might be schizoid, I've been like "...yeah? somewhat? maybe? seems mixed? aren't I just avoidant? but I feel more worried about relating than about being rejected?", though I'm not very motivated to learn a bunch about it. So IDK. But anyway, re/ egosyntonicity:

• Compared to avoidant, schizoid seems vaguely similar, but less pathetic; less needy or cowardly.
• Schizoid has some benefits of "disagreeability, but not as much of an asshole". Thinking for yourself, not being taken in by common modes of being.
• Schizoid is maybe kinda like "I have a really really high bar for who I want to relate to", which is kinda high-status.

I describe myself as a techno-hippy. I was reading Cory Doctorow while waiting for the next chapter of HPMoR. I’ve often wanted a good leftrat community. I feel ya. I have an intuition that Consequentialism is a dangerous philosophy to adopt for optimizers, especially when they get scared. It’s not a huge hop from “saving the world” to “saving people from themselves.”

When I was in my twenties, I’d spend hours a day in forums and on Reddit. I’ve moderated and participated. Now, I’m 47 I just don’t have it in me. Planting a banner works, but it’s a lot of work. 

This is about as close as I get to social media nowadays. It’s too easy to get people angry online, and to use that anger as a weapon. Right now I’m hoping I can guide my son through his teen-age years without him being weaponized. Hopefully kindness and the virtues of Stoicism stand stronger against inversion than Consequentialism. 

 

Two things lead me to think human content online will soon become way more valuable.

1. Scarcity. As AI agents begin fill the internet with tons of slop, human content will be relatively scarcer. Other humans will seek it out.
2. Better routing. As AI leads to the improvement of search/recommendation systems, human content will be routed to exactly the people who will value it most. (This is far from the case Twitter/Reddit today). As human content is able to reach more of the humans that value it, it gets valued more. That includes existing human content: most of the content online that is eerily relevant to you... you haven't seen yet because surfacing algorithms are bad.

The implication: make tons of digital stuff. Write/Draw/Voice-record/etc

when i was new to research, i wouldn't feel motivated to run any experiment that wouldn't make it into the paper. surely it's much more efficient to only run the experiments that people want to see in the paper, right?

now that i'm more experienced, i mostly think of experiments as something i do to convince myself that a claim is correct. once i get to that point, actually getting the final figures for the paper is the easy part. the hard part is finding something unobvious but true. with this mental frame, it feels very reasonable to run 20 experiments for every experiment that makes it into the paper.

2b/2c. I think I would say that we should want a tyranny of the present to the extent that is in our values upon reflection. If, for example, Rome still existed and took over the world, their CEV should depend on their ethics and population. I think it would still be a very good utopia, but it may also have things we dislike.
Other considerations, like nearby Everett branches... well they don't exist in this branch? I would endorse game theoretical cooperation with them, but I'm skeptical of any more automatic cooperation than what we already have. That is, this sort of fairness is a part of our values, and CEV (if not adversarially hacked) should represent those already? I don't think this would end up in a tyranny anything like the usual form of the word if we're actually implementing CEV. We have values for people being able to change and adjust over time, and so those are in the CEV. There may very well be limits to how far we want humanity to change in general, but that's perfectly allowed to be in our values. Like, as a specific example, some have said that they think global status games will be vastly important in the far future and thus a zero-sum resource. I find it decently likely that an AGI implementing CEV would discourage such, because humans wouldn't endorse it on reflection, even if it is a plausible default outcome.

Like, essentially my view is: Optimize our-branches' humanity's values as hard as possible, this contains desires for other people's values to be satisfied, and thus they're represented. Other forms of fairness to things we aren't completely a fans of can be bargained for (locally, or acausally between branches/whatever).

So that's my argument against the tyranny and Everett branches part. I'm less skeptical of considering whether to include the recently dead, but I also don't have a great theory of how to weight them. Those about to be born wouldn't have a notable effect on CEV, I'd believe.

The option you suggest in #3 is nice, though I think it runs some risks of being dominated or notably influenced by "humans in other very odd branches", and so we're outweighed by them despite them not locally existing. I think it is less that you want a human predicate, and more of a "human who has values compatible with this local branch". This is part of why I advocate just bargaining between branches: if the humans in an AGI-made New Rome want us to instantiate their constructed friendly/restricted AGI Gods locally to proselytize, they can trade for it rather than that faction being automatically divvied out a star by our AGI's CEV.
"Human who has values compatible with this local branch" feels weak as a definition, arbitrary, but I'm not sure we can do better than that. I imagine we'd even have weightings, because we likely legitimately value baby's in special ways that don't entail maxing out reward centers or boosting them to megaminds soon after birth, we have preferences about that. Then of course there's minds that are sortof humanish, which is why you'd have a weighting.

(This is kinda rambly, but I do think a lot of this can be avoided with just plain CEV because I think most people on reflection would end up with "reevaluate whether the deal was fair with reflection and then adjust the deal and reference class based on that".)

I like Wentworth's toy model, but I want it to have more numbers, so I made some up. This leads me to the opposite conclusion to Wentworth.

I think (2-20%) is pretty sensible for successful intentional scheming of early AGI.

Assume the Phase One Risk is 10%.

Superintelligence is extremely dangerous by (strong) default. It will kill us or at least permanently disempower us, with high probability, unless we solve some technical alignment problems before building it.

Assume the Phase Two Risk is 99%. Also:

• Spending an extra billion dollars on AI control reduces Phase One Risk from 10% to 5%.
• Spending an extra billion dollars on AI alignment reduces Phase Two Risk from 99% to 98%.

The justification for these numbers is that each billion dollars buys us a "dignity point" aka +1 log-odds of survival. This assumes that both research fields are similarly neglected and tractable.

Therefore:

• Baseline: by default we get 9 milli-lightcones.
• If we spend on AI control we get 9.5 milli-lightcones. +0.5 over baseline.
• If we spend on AI alignment we get 18 milli-lightcones, +9 over baseline.

We should therefore spend billions of dollars on both AI control and AI alignment, they are both very cost-efficient. This conclusion is robust to many different assumptions, provided that overall P(Doom) < 100%. So this model is not really a "case against AI control research".

It has been pretty clearly announced to the world by various tech leaders that they are explicitly spending billions of dollars to produce "new minds vastly smarter than any person, which pose double-digit risk of killing everyone on Earth". This pronouncement has not yet incited riots. I feel like discussing whether Anthropic should be on the riot-target-list is a conversation that should happen after the OpenAI/Microsoft, DeepMind/Google, and Chinese datacenters have been burnt to the ground.

Once those datacenters have been reduced to rubble, and the chip fabs also, then you can ask things like, "Now, with the pressure to race gone, will Anthropic proceed in a sufficiently safe way? Should we allow them to continue to exist?" I think that, at this point, one might very well decide that the company should continue to exist with some minimal amount of compute, while the majority of the compute is destroyed. I'm not sure it makes sense to have this conversation while OpenAI and DeepMind remain operational.

Someone just noted that the Review Voting widget might imply that the "Jan 5" end time is meant to be inclusive of a full 24 hours from now, which wasn't the intent, but given that people may have been expecting that, and that the consequences for Not Extending aren't particularly bad, I'm going to give people another 24 hours.

Meanwhile, I said it elsewhere but will say again here: if that there were any posts you were blocked from commenting on that you want to write a review of... I forgot to fix that in the code and it'll take a little while to fix, but meanwhile you can write a shortform comment and DM me (in which I will manually set it to be a review of that post), and/or write a top level post and tag it with 2023 Longform Reviews [? · GW] (which will cause it to appear in the Frontpage Review widget)