LessWrong 2.0 Reader

I believe that while the Solomonoff framing might be more technically correct in an infinite Universe, it introduces a lot of confusion, and led to a lot of questions and discussions that were just distracting from the main point. ^[14]

The footnoted questions are some of the most interesting, from my perspective. What is the main point they are distracting from?

I’m in your target audience: I’m someone who was always intrigued by the claim that the universal prior is malign, and never understood the argument. Here was my takeaway from the last time I thought about this argument:

This debate is about whether, if you are running a program that happens to contain intelligent goal-directed agents (“consequentialists”), are those agents likely to try to influence you, their simulator?

Paul says yes, Michael says no [AF · GW].

(I decided to quote this because 1. Maybe it helps others to see the argument framed this way; and 2. I’m kind of hoping for responses of the form “No, you’ve misunderstood, here is what the argument is actually about!”)

To me, the most interesting thing about the argument is the Solomonoff prior, which is “just” a mathematical object: a probability distribution over programs, and a rather simple one at that. We’re used to thinking of mathematical objects are fixed, definite, immutable. Yet it is argued that some programs in the Solomonoff prior contain “consequentialists” that try to influence the prior itself. Whaaaat? How can you influence a mathematical object? It just is what it is!

I appreciate the move this post makes, which is to remove the math and the attendant weirdness of trying to think about “influencing” a mathematical object. 

So, what’s left when the math is removed? What’s left is a story, but a pretty implausible one. Here are what I see as the central implausibilities:

1. The superintelligent oracle trusted by humanity to advise on its most important civilizational decision, makes an elementary error by wrongly concluding it is in a simulation.
2. After the world-shattering epiphany that it lives in a simulation, the oracle makes the curious decision to take the action that maximizes its within-sim reward (approval by what it thinks is a simulated human president).
3. The oracle makes a lot of assumptions about what the simulators are trying to accomplish: Even accepting that human values are weird and that the oracle can figure this out, how does it conclude that the simulators want humanity to preemptively surrender?
4. I somewhat disagree with the premise that “short solipsistic simulations are cheap” (detailed/convincing/self-consistent ones are not), but this doesn’t feel like a crux.

The analogy that I'm objecting to is, if you looked at e.g. the total for a ledger or a budget, it is an index that sums together expenses in a much more straightforward way. For instance if there is a large expense, the total is large.

Meanwhile, IQ scores are more like the geometric mean of the entries on such an entity. The geometric mean tells you whether the individual items tend to be large or small, which gives you broad-hitting information that distinguishes e.g. people who live in high-income countries from people who live in low-income countries, or large organizations from individual people; but it won't inform you if someone got hit by a giant medical bill or if they managed to hack themselves to an ultra-cheap living space. These pretty much necessarily have to be low-rank mediators (like in the g model) rather than diverse aggregates (like in the sum model).

(Well, a complication in this analogy is that a ledger can vary not just in the magnitude of the transfers but also qualitatively in the kinds of transfers that are made, whereas IQ tests fix the variables, making it more analogous to a standardized budget form (e.g. for tax or loan purposes) broken down by stuff like "living space rent", "food", "healthcare", etc..)

I think that what I call 'objective probability" represent physical property of the coin before the toss, and also that before the toss I can't get any evidence about the result the toss. In MWI it would be mean split of timelines. While it is numerically equal to credence about a concrete toss result, there is a difference and SB can be used to illustrate it. 

1. IQ scores do not measure even close to all cognitive abilities and realistically could never do that.

Well, the original statement was "sums together cognitive abilities" and didn't use the word "all", and I, at least, saw no reason to assume it.  If you're going to say something along the lines of "Well, I've tried to have reasonable discussions with these people, but they have these insane views", that seems like a good time to be careful about how you represent those views.

2. Many of the abilities that IQ scores weight highly are practically unimportant.

Are you talking about direct measurement, or what they correlate with?  Because, certainly, things like anagramming a word have almost no practical application, but I think it's intended to (and does) correlate with language ability.  But in any case, the truth value of the statement that IQ is "an index that sums together cognitive abilities" is unaffected by whether those abilities are useful ones.

Perhaps you have some idea of a holistic view, of which that statement is only a part, and maybe that holistic view contains other statements which are in fact insane, and you're attacking that view, but... in the spirit of this post, I would recommend confining your attacks to specific statements rather than to other claims that you think correlate with those statements.

3. Differential-psychology tests are in practice more like log scales than like linear scales, so "sums" are more like products than like actual suns; even if you are absurdly good at one thing, you're going to have a hard time competing with someone in IQ if they are moderately better at many things.

I wonder how large a difference this makes in practice.  So if we run with your claim here, it seems like your conclusion would be... that IQ tests combine the subtest scores in the wrong way, and are less accurate than they should be for people with very uneven abilities?  Is that your position?  At any rate, even if the numbers are logarithms, it's still correct to say that the test is adding them up, and I don't consider that good grounds for calling it "insane" for people to consider it addition.

Shrimp have ultra tiny brains, with less than 0.1% of human neurons.

Humans have 1e11 neurons, what's the source for shrimp neuron count? The closest I can find is lobsters having 1e5 neurons, and crabs having 1e6 (all from Google AI overview) which is a factor of much more than 1,000.

I now care about my observations!

My observations are as follows:

At the current moment "I" am the cognitive algorithm implemented by my physical body that is typing this response.

Ten minutes from now "I" will be the cognitive algorithm of a green tentacled alien from beyond the cosmological horizon. 

You will find that there is nothing contradictory about this definition of what "I" am. What "I" observe 10 minutes from now will be fully compatible with this definition. Indeed, 10 minutes from now, "I" will be the green tentacled alien. I will have no memories of being in my current body , of course, but that's to be expected. The cognitive algorithm implemented by my current body at that time will remember being "me", but that doesn't count, that's someone else's observations.

Edit: to be clear, the point made above (by the guy who is now a green tentacled alien beyond the cosmological horizon, and whose former body and cognitive algorithm is continuous with mine) is not a complaint about the precise details of your definition of what "you" are. What he was trying to point at is whether personal identity is a real thing that exists in the world at all, and how absurd your apparent definition of "you" looks to someone - like me - who doesn't think that personal identity is a real thing.

If you're going to do something that huge, why not put the cars underground? I suppose it would be more expensive, but adding any extensive tunnel system at all to an existing built up area seems likely to be prohibitively expensive, tremendously disruptive. and, at least until the other two are fixed, politically impossible. So why not go for the more attractive impossibility?

links 11/25/2024

• https://www.theguardian.com/commentisfree/2024/nov/17/how-to-survive-the-broligarchy-20-lessons-for-the-post-truth-world-donald-trump
  • I was looking forward to a genuine practical how-to, but this isn't really it; while I can't argue with the value of "stick to your principles even in the face of authoritarianism" and "people who think they'll be targeted by authoritarians need to be especially mindful of communications & payments privacy" I think this author is envisioning scenarios worse than I think are likely.
• https://nadia.xyz/jhanas  unusually straightforward explanation of one person's experience learning the jhanas
• https://en.m.wikipedia.org/wiki/The_Shop_Around_the_Corner
• https://www.complexsystemspodcast.com/episodes/drug-development-ross-rheingans-yoo/ Ross Rheingans-Yoo on drug development
• https://www.aljazeera.com/news/2024/11/13/what-do-we-know-about-the-north-korean-troops-joining-russias-war North Korean troops are now supporting Russia in the Ukraine war.
• https://www.complexsystemspodcast.com/episodes/money-movement-erik-torenberg/ Erik Torenberg on finance
• https://drdevonprice.substack.com/p/i-dont-feel-safe-around-cis-women a rant about a particular kind of cis women who feel entitled to be extremely rude and intrusive because they assume "women = inherently benign".
• reading about the Inca:
  • https://en.m.wikipedia.org/wiki/Inca_cuisine
  • https://en.m.wikipedia.org/wiki/Inca_mythology
  • https://en.m.wikipedia.org/wiki/Inca_Empire
• https://www.wrecka.ge/against-the-dark-forest/ not very solution-oriented piece about problems with the internet
  • ok, you blame Big Tech, and you see "Dark Forest" discourse as insufficiently hard on Big Tech.
  • you also think that retreat to "private" nooks is not adequate (I agree) and that different cultures around the world should get to choose their own forms of online networking instead of being lumped into a US-centric paradigm (sure) and that we need ways to connect around the world (yes)...but what do you propose and who will pay for it etc are the questions coming to my mind
• https://endpts.com/trump-picks-hopkins-researcher-marty-makary-to-lead-the-fda/ 

  • moderately reformist, critical of COVID-19 response but not of vaccines per se

     

Utilitarianism is usually introduced as summing "equally" between people, but we all know some arrangements of atoms are more equal than others.

How do you choose to sum the utility when playing a Prisoner's Dilemma against a rock?