LessWrong 2.0 Reader

Oops, just read this response. 

On the intuition question, my intuition was probably the other way because most of human history was non-farming, and because the vast majority of farmers (those born in the last millennium) weren't my ancestors. 

I updated my model to account for an error - it's now a bit closer. 7.8 billion non-farmers to 6.4 billion farmers, and 4.9 billion exclusive farmers, but I still basically stand by the logic.

To respond to your question, why I didn't pick a fixed number of personal ancestors:

We have fewer recent ancestors, assuming 16 generations, we'd have around 20k to 50k ancestors at 1600. (2^16 - inbreeding). If we want to count these ancestors carefully, we should count back with an algorithm accounting for population size and exponentially increasing inbreeding. 

We could also plausibly try to use this strategy to draw a more accurate number of ancestors from 1200-1600--- this might be a period where individual/geographical differences, or population constraints, play a significant role. If you're Icelandic, most of your ancestors in this period will still be from Iceland, but if you are Turkish, your ancestors from this period are more likely to extend from Britain to Japan. My model doesn't do this, because it sounds difficult, and because the numbers are negligible anyway- I just estimate that 0.1% to 1% of total humans born from 1200- today were my ancestors. 

By around 1200 AD, it surely becomes impractical to rely on a personal family tree to track ancestry, because of the exponential growth in the number of ancestors. Beyond that point, your total potential ancestors (in the billions, without factoring in inbreeding) massively exceed the global population (in the 100s of millions). The limited population size becomes the constraint. 

So an Italian might assume that they are descended from a significant portion (40%?) of Europe’s population in 1200 AD. By 800 AD, this would extend to a majority (60%?) of people living across Eurasia and Northern Africa. By the time we reach 500 BC to 1000 AD, it’s likely that most people from the major Old World civilizations and peripheries (where the bulk of the global population lived) were direct ancestors of people alive today. My numbers could be way off, but I think this is a better way of getting in the right ballpark than trying to trace back individual ancestry. I used these figures as a baseline. https://www.prb.org/articles/how-many-people-have-ever-lived-on-earth/

You're right that I don't account for major bottlenecks - my assumption is that they basically even out over time, and there's a constant 20-60% chance of humans born in each period not passing down ancestors to the modern day. If you wanted to refine this model you'd take into account more recent (e.g. Black Death) and less recent (Neolithic Y-chromosome bottleneck) bottlenecks.

Button is very intimidating, indeed.

Yeah I'm on board with deontological-injunction shaped constraints. See here [LW · GW] for example.

Perhaps instead of "attempting to achieve the goal at any cost" it would be better to say "being willing to disregard conventions and costs imposed on uninvolved parties, if considering those things would get in the way of the pursuit of the goal".

Nah I still disagree. I think part of why I'm interpreting the words differently is because I've seen them used in a bunch of places e.g. the lightcone handbook to describe the lightcone team. And to describe the culture of some startups (in a positively valenced way).

Being willing to be creative and unconventional -- sure, but this is just part of being capable and solving previously unsolved problems. But disregarding conventions that are important for cooperation that you need to achieve your goals? That's ridiculous. 

Being willing to impose costs on uninvolved parties can't be what is implied by 'going hard' because that depends on the goals. An agent that cares a lot about uninvolved parties can still go hard at achieving its goals.

I suspect we may be talking past each other here.

Unfortunately we are not. I appreciate the effort you put into writing that out, but that is the pattern that I understood you were talking about, I just didn't have time to write out why I disagreed.

I expect this to continue to be true in the future

This is the main point where I disagree. The reason I don't buy the extrapolation is that there are some (imo fairly obvious) differences between current tech and human-level researcher intelligence, and those differences appear like they should strongly interfere with naive extrapolation from current tech. Tbh I thought things like o1 or alphaproof might cause the people who naively extrapolate from LLMs to notice some of these, because I thought they were simply overanchoring on current SoTA, and since the SoTA has changed I thought they would update fast. But it doesn't seem to have happened much yet. I am a little confused by this.

What observations lead you to suspect that this is a likely failure mode?

I didn't say likely, it's more an example of an issue that comes up so far when I try to design ways to solve other problems. Maybe see here [LW · GW] for instabilities in trained systems, or here for more about that particular problem.

I'm going to drop out of this conversation now, but it's been good, thanks! I think there are answers to a bunch of your claims in my misalignment and catastrophe [LW · GW] post.

I'm thinking that the most ethical (morally least risky) way to "insure" against a scenario in which AI takes off and property/wealth still matters is to buy long-dated far out of the money S&P 500 calls. (The longest dated and farthest out of the money seems to be Dec 2029 10000-strike SPX calls. Spending $78 today on one of these gives a return of $10000 if SPX goes to 20000 by Dec 2029, for example.)

My reasoning here is that I don't want to provide capital to AI industries or suppliers because that seems wrong given what I judge to be high x-risk their activities are causing (otherwise I'd directly invest in them), but I also want to have resources in a post-AGI future in case that turns out to be important for realizing my/moral values. Suggestions welcome for better/alternative ways to do this.

I contend that P(H2) is very close to P(H1), and certainly in the same order of magnitude, since (conditional on H1) a simulation that does not test for H2 is basically useless.

As for priors I’d refuse to update down – well, the ASI is smarter than either of us!

For the principle of indifference to apply, you’d need infinitely many purposes as plausible as this one, or at least similarly plausible. I can’t imagine how this could hold. Can you think of three?

But this assumes a model should aim to fit all data, which is a waste of effort.

Being pets also means human agency would no longer be a relevant input to the trajectory of human lives.

You aren't supposed to use metaethics to settle ethical arguments, the point of metaethics is to get people to stop discussing metaethics.

I greatly enjoyed this book back in the day, but the whole scenario was wild enough to summon the moral immune system. Past a certain point, for me it’s a safe default to put up mental barriers and actively try not to learn moral lessons from horror fiction. Worm, Gideon the 9th, anything by Stephen King- great, but I don’t quite expect to learn great lessons.

While rejecting them as sources of wisdom now, I can remember these books and return to them if I suddenly need to make moral choices in a world where people can grow wiser by being tortured for months, or stronger by