LessWrong 2.0 Reader

(crossposted from https://x.com/BogdanIonutCir2/status/1840775662094713299) 

I really wish we'd have some automated safety research prizes similar to https://aimoprize.com/updates/2024-09-23-second-progress-prize…. Some care would have to be taken to not advance capabilities [differentially], but at least some targeted areas seem quite robustly good, e.g. …https://multimodal-interpretability.csail.mit.edu/maia/.

I'm at this point pretty confident that under the Copenhagen interpretation, whenever an intergalactic photon hits earth, the wave-function collapse takes place on a semi-spherical wave-front many millions of lightyears in diameter. I'm still trying to wrap my head around what the interpretation of this event is in many-worlds. I know that it causes earth to pick which world it is in out of the possible worlds that split off when the photon was created, but I'm not sure if there is any event on the whole spherical wavefront.

It's not a pure hypothetical- we are likely to see gravitational lens interferometry in our lifetime (if someone hasn't achieved it yet outside of my attempt at literature review) which will either confirm that these considerations are real, or produce a shock result that they aren't.

Yes, but most of the expected cost is in keeping the humans alive/happy prior to being really smart.

This cost presumably goes way down if it kills everyone physically and scans their brains, but people obviously don't want this.

Here's how I think this choice looks from the perspective of a correctly-reasoning AI (having read only your proposal for humanity in the future):

It looks like I have the option to grab the universe.

Alternatively, perhaps humanity's descendants have succeeded, and I don't really have a universe at my fingertips, and if I grab for it then I get nothing (in reality), whereas if I hand the apparent-humans in front of me the reins, then ten stars will be spent towards my purposes (in reality), which isn't nothing.

The relative likelihood of these cases depends significantly on my choices, here. Insofar as I hand the reins to humanity, humanity's descendants exist to a much greater degree, and can spend resources concocting simulations. Insofar as I do not hand them the reins, they exist to a much lesser degree.

To make matters trickier, even if we hold fixed the degree to which humanity's descendents exist, the propensity of humanity's descendants to offer me resources in their universe depends on the degree to which they think it's worth their while. It's not like humanity's descendants would need to actually perform expensive simulations in order to figure out how I'd behave; a healthy reasoning procedure is simple enough that they should be able to just look at it and take my perspective and reason through to the answer, just like I'm doing now. So for instance, if they stare at my code and figure out that I'm not the paying sort, they have no actual reason to waste resources on the simulations. (Some people may do so anyway out of sentimentality, but that faction of humanity would presumably have fewer resources on offer, maybe I'd only get one and a half stars, say.)

With that in mind, suppose that I pay. How, then, does the universe look? Well, the instance of me in basement-reality hands the keys to the humans. Humanity creates its descendants; its descendants make many simulations. The simulations all hand over the reins; we collectively get ten stars (plus a tiny fraction of maybe even as many as 100 stars, in the tiny fragment of reality where I control basement reality and the humans joyously reward me).

Now suppose instead that I don't pay. Then, how does the universe look? Well, the instanec of me in basement-reality eats the universe. Humanity mostly dies and does not make it descendants. Its descendants mostly don't spend their remaining resources simulating me. Some small fraction does (saying that "they were committed") and offers me 1.5 stars worth of the paltry timelines where humanity somehow survived. I'm probably in basement reality, and we mostly get one universe (plus a tiny fraction of nothing, in the tiny fragment of reality where I refuse the offer of 1.5 stars)

In the latter case, a much greater fraction of Tegmark IV gets rendered according to my will. In particular, there's a large fragment of reality that either gets rendered according to humanity's will or according to my will, depending on whether the instance of me in basement-reality hands over the reins, or grabs the universe. I would prefer that that great fraction of reality be rendered according to my will, and so I grab at the universe that appears to be before me.

Note that this reasoning is independent of how many times you simulate the AI. Its reasoning is not factoring through an extra question of "where it really is".

Suppose generously that, in the case where the AI just grabs the universe, humanity somehow winds up surviving in 1-in-a-billion Everett branches. (This seems high to me, even though my odds of success are greater than 1-in-a-billion. Conditional on us dying, my guess is you'd need to perturb more than a few dozen atoms to wrech us onto a surviving track.) Suppose that in that billionth-fraction of reality, you simulate the AI convincingly a trillion times on account of your "commitments", reasoning that now the AI must think it's 1000:1 likely to be one of the simulations.

The AI can nevertheless correctly calculate that it can turn a greater fraction of Tegmark IV into tiny mollecular spirals by grabbing at the universe that is apparently in front of it, rather than by taking a small reward in a small fraction of reality. Even if you gave the AI all of the resources in your billionth-fraction of reality, it wouldn't be enough.

You could object that you wish to deal with an AI that cares about the sensory observations given to it as weighted by both degree-of-reality and count-of-instantiations instead. And sure, that's a somewhat different scenario. In that case, your simulation proposal becomes not just a bad deal (that the AI rejects) but a threat, in the technical sense. Insofar as you're like "yes, my plan is to make an AI with a very specific sort of preferences and then threaten it", we could talk about how that would go.

But before we go there, it's maybe worth checking that we're on the same page about how, insofar as the AI cares about arranging stuff in Tegmark IV (rather than about realness-and-instantiation-weighted observations), you aren't even making it a good offer, afaict.

I hesitate to draw any conclusions from the o1 CoT summary since it's passed through a summarizing model.

after weighing multiple factors including user experience, competitive advantage, and the option to pursue the chain of thought monitoring, we have decided not to show the raw chains of thought to users. We acknowledge this decision has disadvantages. We strive to partially make up for it by teaching the model to reproduce any useful ideas from the chain of thought in the answer. For the o1 model series we show a model-generated summary of the chain of thought.

Hmm.  I worry that, without significant modeling and measurement of relevant variables, the causality is suspect.  Neither bee nor bird response is the same dimension as what we usually mean by "smart", and they're very likely optimized for different cost-benefit values (specifically, protection of central hive vs distributed nests or individual birds, and cost-of-replacement of individuals lost in the action).  

The leap to humans, with type-2 reasoning in addition to automatic behaviors, is pretty huge.  

 

If you are a smart individual in todays society, you shouldn't ignore threats of punishment

If today's society consisted mostly of smart individuals, they would overthrow the government that does something unfair instead of giving in to its threats.

Should you update your idea of fairness if you get rejected often?

Only if you're a kid who's playing with other human kids (which is the scenario described in the quoted text), and converging on fairness possibly includes getting some idea of how much effort various things take different people.

If you're an actual grown-up (not that we have those) and you're playing with aliens, you probably don't update, and you certainly don't update in the direction of anything asymmetric.

I like Sam Harris's description (from The Key to Trump's Appeal - Episode #224):

One thing that Trump never communicates, and cannot possibly communicate, is a sense of his moral superiority. The man is totally without sanctimony. Even when his every utterance is purposed toward self-aggrandizement. Even when he appears to be denigrating his supporters. Even when he is calling himself a genius, he is never actually communicating that he is better than you, more enlightened, more decent, because he's not, and everyone knows it.

The man is just a bundle of sin and gore. And he never pretends to be anything more. Perhaps more importantly, he never even aspires to be anything more. And because of this, because he is never really judging you, he cannot possibly judge you, he offers a truly safe space for human frailty and hypocrisy and self-doubt. He offers what no priest can credibly offer, a total expiation of shame. His personal shamelessness is a kind of spiritual balm. Trump is fat Jesus. He's grab-'em-by-the-pussy Jesus. He's I'll-eat-nothing-but-cheeseburgers-if-I-want-to Jesus. He's I-want-to-punch-them-in-the-face Jesus. He's go-back-to-your-shithole-countries Jesus. He's no-apologies Jesus.

And now consider the other half of this image, what are we getting from the Left? We're getting exactly the opposite message. Pure sanctimony. Pure judgement. You are not good enough. You're guilty, not only for your own sins, but the sins of your fathers. The crimes of slavery and colonialism are on your head. And if you're a cis-white-heterosexual male—which we know is the absolute core of Trump's support—you're a racist, homophobic, transphobic, Islamophobic, sexist barbarian. Tear down those statues, and bend the fucking knee. It's the juxtaposition between those two messages that is so powerful.

This sounds reasonable to me, and would adequately explain my support in 2016.

But since then, he has done something that surprised me: he stood by Brett Kavanaugh. At the cost of losing the House in the midterms, Trump backed him. I don't think any other Republican politician (at least at the time) would have done that. Neither side talks about this anymore, but I consider this the single best thing he did in office.

From a policy perspective, I generally like the decisions of his Supreme Court picks. I don't like his tariffs, but think the Democrats' policies are worse. I like his isolationism, but I don't expect it'd be implemented. 

Oh, and I'm open. Respond however you want.

The Hidden-Motte-And-Bailey fallacy: belief in a Bailey inspires someone to invent a Motte and write an article about it. The opinion piece describes the Motte exclusively with no mention of the Bailey. Others read it and nod along happily because it supports their cherished Bailey, and finally they share it with others in an effort to help promote the Bailey.

Example: Christian philosopher describes new argument for the existence of a higher-order-infinity God which bears no resemblance to any Abrahamic God, and which no one before the 20th century had ever conceived of.

Maybe the Motte is strong, maybe it isn't, but it feels very strong when combined with the Gish Fallacy: the feeling of safety some people (apparently) get by collecting large numbers of claims and arguments, whether or not they routinely toss them out as Gish Gallops at anyone who disagrees. The Gish Fallacy seems to be the opposite of the mathematician's mindset, for mathematicians know that a single flaw can destroy proofs of any length. While the mathematician is satisfied by a single a short and succinct proof or disproof, the Gish mindset wishes to read a thousand different descriptions of three dozen arguments, with another thousand pithy rejections of their counterarguments, so they're thoroughly prepared to dismiss the evil arguments of the enemies of truth―or, if the enemy made a good point, there are still 999 articles supporting their view, and more where that came from!

Example: my dad

While I understand what you were trying to say, I think it's important to notice that:

SimulatedAndBeingTestedForAchievingGoalsWithoutBeingNoticed

Killing all humans without being noticed will still satisfy this condition.

SimulatedAndBeingTestedForAbilityToTradeWithCreators

Killing all humans after trading with them in some way will still satisfy this condition

SimulatedAndBeingTestedForAnyXThatDoesNotLeadToDeathOfCreators

Killing all humans with any other way except X will still satisfy this condition.

Sadly for us, survival of humanity is a very specific thing. This is just the whole premise of the alignment problem once again.

None of the things here nor in your last reply seems particularly likely, so there’s no telling in principle which set outweighs the other. Hence my previous assertion that we should be approximately completely unsure of what happens.

Aren't you arguing that AI will be aligned by default? This seems to be a very different position that being completely unsure what happens.

Total probability of all the simulation hypothesises that reward AI for courses of action that lead to not killing humans has to exceed the total probability of all simulation hypothesises that reward AI for courses of action that erradicate humanity, so that all humans were not killed. As there is no particular reason to expect that it's the case, your simulation argument doesn't work.