LessWrong 2.0 Reader

By way of analogy:

Say you have the ability to build a box that if someone turns the crank, there’s some non-zero probability of a golden egg popping out. The more you turn the crank, the more likely an egg pops out.

So you build the box, turn the crank, and after a very long time and lots of cranking out pops an egg. Who owns the egg? Well, pretty obviously you, you built the box, you turned the crank. Both activities required the use of your body and other elements, and no one else contributed.

Now, in your old age, your arms are getting tired, but you now rely on the income from selling the eggs. So you have your sons turn the crank alternately and out pops an egg. Who owns the egg? Well, you built the box and turned the crank any number of times, but now the labor of your sons caused the egg to pop out, not yours.

This is complex. Historically and “intuitively” you still own the egg. It’s your box, you’ve been using it your whole life. But is this really intuitive? Not only did your creative work and faithful maintenance cause the box to exist, but the labor of your sons also caused the egg to pop out.

Now you live for a long time arraigning to share the proceeds with your sons for their labor. Eventually you die, the gold notwithstanding. Now your sons, dependent on the egg income, agree to share the box as they have been already for years. They turn the crank, alternately, and eggs pop out. Who owns the eggs? Clearly either of the brothers own some portion of each egg, but how much?

We continue on in this way over centuries with many boxes, many eggs, many creators, and many laborers.

Now the question is, who owns the boxes, and who owns the eggs?

In my previous post I gave a theory that the crank turners own the box and the eggs in proportion to their crank turning, but perhaps this treats the box creators unfairly. What do you think?

Nice!

For the 2024 prediction "So, the most compute spent on a single training run is something like 5x10^25 FLOPs." you cite v3 as having been trained on 3.5e24 FLOP, but that is outside an OOM. Whereas Grok-2 was trained in 2024 with 3e25, so seems to be a better model to cite?

Re democratic countries overtaken by dictatorial countries, I think that this will only last until AI that can automate at least all white collar labor is achieved, and maybe even most blue collar physical labor well enough that human wages for those jobs decline below what you need to subsist on a human, and by then dictatorial/plutocratic countries will unfortunately come back as a viable governing option, and maybe even overtaking democratic countries.

So to come back to the analogy, I think VNM-rationality dictatorship is unfortunately common and convergent over a long timescale, and it's democracies/coalition politics that are fragile over the sweep of history, because they only became dominant-ish starting in the 18th century and end sometime in the 21st century.

I haven't tried to calculate it recently. I still feel rather pessimistic for all the usual reasons. This paper from OpenAI was a positive update; Vance's strong anti-AI-safety stance was a negative update. There have been various other updates besides. 

I believe it is a clear demonstration that misalignment likely does not stem from the model being “evil.” It simply found a better way to achieve its goal using unintended means.

It is fascinating to see that the official science has finally discovered what Yudkowsky wrote about a decade ago. Better late than never, I guess.

Surely it would be exceptionally good at those kinds of writing, too, right?

surely an LLM capable of writing A+ freshman college papers would correctly add two 2-digit numbers?  surely an AI capable of beating grandmasters in chess would be able to tutor a 1000 elo player to a 1500 elo or beyond?  surely an AI capable of answering questions at a university level in diverse subjects such as math, coding, science, law, would be able to recursively improve itself and cause an intelligence explosion?  surely such an AI would at least be able to do a simple task like unload a dishwasher without breaking a dish?

i think surely it should be obvious to anyone who's able to mull it through for a few seconds that intelligence does not need to progress along the same paths as it does for human civilization over centuries or for human beings through child development, or even among proto-intelligent animals on earth.  it is surely obvious to me that AI can exhibit surprising mixes of general and non-general intelligence, and that we're not really sure why it works.  there is really no requirement i have left, that before the AI turns us into paperclips, it must 100% be able to beat poker players at the WSOP or generate an oscar-winning feature film, or be able to make nobel-winning science discoveries.  some of these requirements seem more plausible than others, but none seem totally certain.

I disagreed with Gwern at first. I'm increasingly forced to admit there's something like bipolar going on here, but I still think we're also missing something - his cognitive state seems pretty steady month to month, rather than episodes of mania alternating with lucidity.

Someone claimed the latest Musk biography said he was much more normal early in the morning, and much crazier late at night. I need to read the biography and see if that's actually in there; if so, maybe there could be a case for ultradian or ultra-rapid-cycling or something. This could potentially just look like a random mix of good and bad decisions depending on when in the cycle he's making them, with the cycle itself too fast to notice on the scale of news reports. As you say, presumably something about that changed the past few years (I've never heard anyone discuss what happens to ultradian bipolar if you simply never sleep, but I bet it's nothing good).

Anatoly Karlin, who apparently also read the biography, says that Musk's father Errol also went crazy after fifty - see https://x.com/powerfultakes/status/1892003738929238408 . One excerpt:

"I don't know how he went from being great at engineering to believing in witchcraft", [Elon told the biographer about his father]. Errol can be very forceful and occasionally convincing. "He changes reality around him," [Elon's brother] Kimbal  says. "He will literally make up things, but he actually believes his own false reality."

I can't think of a form of bipolar which consistently gets much worse at age 50, but I hope to look into this further.

I don't think that undecidability of exact comparison (as opposed to comparison within any given margin of error) is necessarily a bug, however, if you really want comparison for periodic sequences, you can insist that the utility function is defined by a finite state machine. This is in any case already a requirement in the bounded compute version [LW(p) · GW(p)].

Last night’s comment was much too stream of thought. While much of it is true from a certain perspective it omits the theory of ownership supporting all of it.

We start with founding ownership in a certain way: the basis of ownership is self-ownership. I as a nervous system own my own body because of innervation, and we say that ownership itself is innervation.

This is intuitive. All our upset at violations flows from this sense of ownership of our own bodies. As people, we naturally extend this ownership to things we habitually use. They form extensions of ourselves, that even though we don’t enervate them, our operative control of them in proportion to how often and how effectively we do control them, give us a similar sense of ownership over even if it is not ownership in fact.

Understanding this, we can evaluate property claims. They are merely the claim that “I use that often, and it doesn’t object”, to some degree. This is common sense, and to some degree normative, since no one wants their habitual life supports taken away, just like no one wants their body violated.

If we see this, then we see there is in fact an ownership gradient flowing out from individuals and interacting with other’s gradients, and this manifold is the true ownership of everything that is not innervated.

Understood this way, we have all we need to query historical/legal property claims. All we need ask is who has some habitual interactive use of a given element. Apply this to the office building, and it’s obvious that the workers have the most claim. However, this is still on balance with the owner-of-record’s use, however problematic, as their habitual use supports their own life.

This tension is what we negotiate through the SEC transition and lease to mortgage change. Through the SEC transition, former owners receive support for their life, receiving family heirlooms and any other real property (real property being those things that no one but them habitually use). Through the lease to mortgage change, the true owners of the home by virtue of living there receive that acknowledgement and legal status, while those with fiat ownership through whatever economic coercion receive mortgage payments to ensure that to whatever extent they rely on those payments, they can transition to something founded in real ownership and not exploitation. [Edit: looking at this further, I’m not sure even these are helpful. No one should have to pay 60PMI a month to their former landlord just because they happened to accumulate more currency in the former system. It would also be rewarding those former owners for their exploitation instead of giving them a new start with everyone else.]

However, this is mostly beside the point, to the extent that these economic relationships from former owners do not have any true habitual use claims other than monetary enrichment. The Social Economy already equalizes purchasing power, so no one can claim that a monetary shortfall affects them more than anyone else. To make that claim would be to plead addiction to a certain lifestyle, and there are many other better ways to deal with that.

As I see it, there are two possible explanations for a fine-tuned universe:

• an insane amount of luck (or something other than luck, such as a supernatural force)
• multiverse

The argument for multiverse is precisely that it does not require the insane amount of luck.

Your argument is that with the insane amount of luck, the multiverse is not necessary.

That is correct, but then we are back to the original question: why did we get so extremely lucky?