LessWrong 2.0 Reader

Sounds like you've written the bottom line of wanting to prove your status to others before you found the excuse of it being important to people to know not to mess with you.

We should not give more status to those who determine the fate of the world because then it will be determined not by those that seek to optimize it but those that seek status.

To measure soul you must pick some anthropic definition. If it's "who is most desperately-for-precision modeled by onlookers?", to measure it you acquire omnipotence, model all the onlookers, and check who they look at.

It may help to visualize this graph with the line for Platt's Law drawn in.

Overall I find the law to be pretty much empirically validated, at least by the standards I'd expect from a half in jest Law of Prediction.

My model is something like:

• For any given algorithm, e.g. SVMs, AlphaGo, alpha-beta pruning, convnets, etc., there is an "effective compute regime" where dumping more compute makes them better. If you go above this regime, you get steep diminishing marginal returns.
• In the (relatively small) regimes of old algorithms, new algorithms and old algorithms perform similarly. E.g. with small amounts of compute, using AlphaGo instead of alpha-beta pruning doesn't get you that much better performance than like an OOM of compute (I have no idea if this is true, example is more because it conveys the general gist).
• One of the main way that modern algorithms are better is that they have much large effective compute regimes. The other main way is enabling more effective conversion of compute to performance.
• Therefore, one of primary impact of new algorithms is to enable performance to continue scaling with compute the same way it did when you had smaller amounts.

In this model, it makes sense to think of the "contribution" of new algorithms as the factor they enable more efficient conversion of compute to performance and count the increased performance because the new algorithms can absorb more compute as primarily hardware progress. I think the studies that Carl cites above are decent evidence that the multiplicative factor of compute -> performance conversion you get from new algorithms is smaller than the historical growth in compute, so it further makes sense to claim that most progress came from compute, even though the algorithms were what "unlocked" the compute.

For an example of something I consider supports this model, see the LSTM versus transformer graphs in https://arxiv.org/pdf/2001.08361.pdf

I'm not sure what you mean by 'astronomical waste or astronomical suffering'.  Like, you are writing that everything forever is status games, ok, sure, but then you can't turn around and appeal to a universal concept of suffering/waste, right?

Whatever you are worried about is just like Gandhi worrying about being too concerned with cattle, plus x years, yeah?  And even if you've lucked into a non status games morality such that you can perceive 'Genuine Waste' or what have you...surely by your own logic, we who are reading this are incapable of understanding, aside from in terms of status games.

To be practically immortal, an entity must possess the following qualities:

• a) it must be distributed across many semi-autonomous nodes; killing some nodes should not cause the death of the entity itself
• b) the nodes must be distributed across vast distances; no natural disaster should be able to kill all the nodes
• c) the entity should be able to create its own backups, hidden in many remote places; no system-level sickness of the whole entity should stop it from being restored from backups
• d) the entity should be able to rapidly self-improve itself, to win against intelligent adversaries

If we consider only biological entities, the supercolony of ants in Southern Europe is the closest to being immortal. It consists of billions of semi-autonomous nodes distributed across a 6 000 km stretch of land. Of course, it still doesn't possess all the 4 qualities, and thus is still mortal. 

Although humans consist of individual nodes too, the described path to biological immortality was closed for them a long time ago. There is no way for the natural selection to take a human and gradually convert it into something like an ant colony. 

And that's the main reason why humans are not immortal (yet): our monkey bodies are not designed for immortality, and cannot be redesigned for that by the natural selection. 

On the other hand, mind uploading could give us all the qualities of immortality, as listed above. 

An entity with the described qualities cannot be out-competed by your "Horde of Death", as such an entity can simply sit and wait until the Horde becomes extinct, which is inevitable for all biological species. 

This is a nice idea. I think it'd need alterations before it became a useful tool (if I'm understanding clearly, and not missing applications of the unaltered version), but it has potential.

[[Note: I haven't looked in any detail at tailcalled's comments/post, since I wanted to give my initial impressions first; apologies for any redundancy]]

Thoughts:

1. There's an Anna Karenina issue: All happiness-inducing AI policies are alike; each unhappiness-inducing policy induces unhappiness in its own way.
   In some real-world situation, perhaps there are ${10}^{30}$ good AI policies and ${10}^{50}$ bad ones. A corrigibility measure that can be near-maximized by allowing us complete control over which of the bad policies we get (and no option to get a good policy) isn't great. Intuitively, getting to choose among any of the good policies is much more corrigible than getting to choose among any of the bad ones - but as things stand (IIUC), $Corrigibilit{y}_{PM}$ says that the former is much less corrigible than the latter.
   I think there needs to be some changes-we-actually-care-about weighting. Probably that looks like bringing in human utility and/or a prior over which changes the human might want.
2. I think we need something more continuous than a set of "cognitively accessible" policies. This should allow dealing with manipulation that's shifts human policies to be more/less accessible.
   Perhaps it makes sense to put this all in terms of the expected utility cost in [finding and switching to a policy] - i.e. accessibility = 1/(expected cost).
   So e.g. a policy that takes much longer to find is less accessible, as is one which entails the human shooting themselves in the foot (perhaps it's cleanest to think of the former as first picking a [search-for-a-better-policy] policy, so that there's no need to separate out the search process).

I suppose you preferred not to involve expected utility much (??), but I think in not doing so you end up implicitly assuming indifference on many questions we strongly care about. (or rather ending up with a measure that we'd only find useful if we were indifferent on such questions)

1: To me, it made it more entertaining and thus easier to read. (No idea about non-anecdotal data, would also be interested.)

3: Also no data; I strongly suspect the metric is generally good because... actually I think it's just because the people I find worth listening to are overwhelmingly not condescending. This post seems highly usual in several ways.

I've gotten one private message expressing more or less the same thing about this post, so I don't think this is a super unusual reaction.

I meant experimentally test to prove to the soul-impoverished that you have more soul than normal. Kind of pointless to be the protagonist if the side characters aren't convinced of your status. It's vital to be able to accomplish this. A government's plans could be spontaneously affected if they hurt the protagonist. By knowing who the protagonist is, a government can more efficiently take action. 

I've never heard frame control used that way despite being fairly familiar with the modern NLP literature. First page of Google search also seems to mostly talk about controling other people's frames.