LessWrong 2.0 Reader

I ask partly because I personally would be more excited of a version of this that wasn't ignoring AGI timelines, but I think a version of this that's not ignoring AGI timelines would probably be quite different from the intellectual spirit/tradition of FHI.

This frame feels a bit off to me. Partly because I don’t think FHI was ignoring timelines, and I think their work has proved quite useful already—mostly by substantially improving our conceptual vocabulary for reasoning about things like existential risk.

But also because it feels important to me to stress that the portfolio of alignment research with maximal expected value need not necessarily perform well in the most likely particular world. One might imagine, for example—and indeed this is my own bet—that the most valuable actions we can take will only actually save us in the subset of worlds in which we have enough time to develop a proper science of alignment.

Yeah, that might be a good idea in case any rich employers stumble on this :-)

In terms of goals, I like making something, having many people use it, and getting paid for it. I'm not as motivated by meaning, probably different from most EAs in that sense.

In terms of skillset, I'd say I'm a frontend-focused generalist. The most fun programming experience in my life was when I built an online map just by myself - the rendering of map data to png tiles, the serving backend, the javascript for dragging and zooming, there weren't many libraries back then - and then it got released and got hundreds of thousands of users. The second most fun was when I made that game - the idea, iterating on the mechanics, graphic design, audio programming, writing, packaging for web and mobile, the whole thing - and it got quite popular too. So that's the prototypical good job for me.

I don't really understand your approach yet. Let's call your decision theory CLDT. You say counterfactuals in CLDT should correspond to consistent universes. For example, the counterfactual "what if a CLDT agent two-boxed in Newcomb's problem" should correspond to a consistent universe where a CLDT agent two-boxes on Newcomb's problem. Can you describe that universe in more detail?

I think you are slightly muddling your phrases.

You are richer if you can afford more goods and better goods. But not all goods will necessarily change price in the same direction. Its entirely possible that you can become richer, but that food prices grow faster than your new income. (For example, imagine that your income doubles, that food prices also double, but prices of other things drop so that inflation remains zero. You can afford more non-food stuff, and the same amount of food, so you are richer overall. This could happen even if food prices had gone up faster than your income.)

I think a (slightly cartoony) real life example is servants. Rich people today are richer than rich people in Victorian times, but fewer rich people today (in developed countries) can afford to have servants. This is because the price of hiring servants has gone up faster than the incomes of these rich people. So it is possible for people to get richer overall, while at the same time some specific goods or services become less accessible.

Maybe a more obvious example is rent (or housing in general). A modern computer programmer in Silicon valley could well be paying a larger percentage of their income on housing than a medieval peasant. But, they can afford more of other things than that peasant could.

After 5 years, I think experience matters more.

Given the state of AI, I think AI systems are more likely to infer our ethical intuitions by default.

You’re basically talking about the software industry. Meta isn’t special. Considering how big the video game industry is, not to mention digital entertainment, and business software, I don’t think we have anything to worry about there.

There is a metaverse already. It's called Second Life and has been around for more than 20 years. Never huge, but never going away. It has a marketplace of virtual goods that residents of Second Life have created. The market deals in "Linden dollars", which can be both bought with real dollars and sold for real dollars.

But look at a few random prices at that Marketplace link. The exchange rate is stable at about L$250 = $1. A skirt for L$399 = $1.60. A massage table (with built-in animations) for L$1698 = $7. (Three times that for the version with built-in sex animations.) A tattoo for L$299 = $1.20. The most expensive car currently on the marketplace is L$50,000 = $200, but there are also plenty selling for under $1.

There are only a very few people who have made a living from selling things in Second Life. The number of spectacular successes might be countable on the fingers of one finger.

While I love Second Life, I do not see an economy of this sort growing to become a substantial part of the total economy. What, after all, is the value of these digital goods? They are decoration for an immersive social space, and game assets for recreational use within that space. They do have value, but the marketplace shows what that value is: $200 for a top-end virtual car.

Suppose someone draws a "personal identity" line to exclude this future sunrise-witnessing person.  Then if you claim that, by not anticipating, they are degrading the accuracy of the sunrise-witness's beliefs, they might reply that you are begging the question.

I disagree. In verbal space MARS and MATS are very distinct, and they look different enough to me.

However, if you want to complain, you should talk to the organisers, not one of the participants.

Here is their website: MARS — Cambridge AI Safety Hub

(I'm not involved in MARS in any way.)