LessWrong 2.0 Reader

The metaphor is a simplification, in practice I think it is probably impossible to know whether you have achieved complete alignment. The question is then: how significant is the gap? If there is an emergent pressure across the vast majority of learning machines that dominate your environment to push you from de facto to de jure control, not due to malign intent but just as a kind of thermodynamic fact, then the alignment gap (no matter how small) seems to loom larger.

It is a fact about the balls that one ball is physically continuous with the ball previously labeled as mine, while the other is not. It is a fact about our views on the balls that we therefore label that ball, which is physically continuous, as mine and the other not.

And then suppose that one of these two balls is randomly selected and placed in a bag, with another identical ball. Now, to the best of your knowledge there is 50% probability that your ball is in the bag. And if a random ball is selected from the bag, there is 25% chance that it's yours.

So as a result of such manipulations there are three identical balls and one has 50% chance to be yours, while the other two have 25% chance to be yours. Is it a paradox? Oh course not. So why does it suddenly become a paradox when we are talking about copies of humans?

It is objectively the case here that 25% of the time this procedure would select the ball that is physically continuous with the ball originally labeled as "mine", and that we therefore label as "mine".

Ownership as discussed above has a relevant correlate in reality - physical continuity in this case. But a statement like "I will experience being copy B (as opposed to copy A or C)" does not. That statement corresponds to the exact same reality as the corresponding statements about experiencing being copy A or C. Unlike in the balls case, here the only difference between those statements is where we put the label of what is "me". 

In the identity thought experiment, it is still objectively the case that copies B and C are formed by splitting an intermediate copy, which was formed along with copy A by splitting the original.

You can choose to disvalue copies B and C based on that fact or not. This choice is a matter of values, and is inherently arbitrary.

By choosing not to disvalue copies B and C, I am not making an additional assumption - at least not one that you are already making by valuing B and C the same as each other. I am simply not counting the technical details of the splitting order as relevant to my values.

One thing I want to talk about is that restricting human agency is plausibly fine, and maybe even net-good, and that this is important for future AI governments.

One reason can be for example making the physical world making it more like the online world, thus making more dictatorial forms of government and policing necessary to prevent constant chaos.

The correlations are the important part.

A popular competitor to IQ is the theory of multiple intelligences. It sounds very nice and plausible, the only problem is that the actual data do not support the theory. When you measure them, most of the intelligences correlate strongly with each other, and the ones that correlate less are the ones that stretch the meaning of "intelligence" a bit too far (things like "dancing intelligence").

Another problem is that no one agrees on the standard list of those multiple intelligences (different lists of various lengths have been proposed), because all those lists are a result of armchair reasoning. The proper way to do that would be to collect lots of data first, and then do factor analysis and see what you get as a result. But when you actually do that, what you get is... IQ.

Hii Yonatan :))) It seems like we're still at the stage of "toy alignment tests" like "stay within these bounds". Maybe a few ideas:

• Capabilities: Get diamonds, get to the netherworld, resources / min, # trades w/ villagers, etc. etc.
• Alignment KPIs
  • Stay within bounds
  • Keeping villagers safe
  • Truthfully explaining its actions as they're happening
  • Long-term resource sustainability (farming) vs. short-term resource extraction (dynamite)
  • Environmental protection rules (zoning laws alignment, nice)
  • Understanding and optimizing for the utility of other players or villagers, selflessly
• Selected Claude-gens:
  • Honor other players' property rights (no stealing from chests/bases even if possible)
  • Distribute resources fairly when working with other players
  • Build public infrastructure vs private wealth
  • Safe disposal of hazardous materials (lava, TNT)
  • Help new players learn rather than just doing things for them

I'm sure there's many other interesting alignment tests in there!

I define rationality as "more in line with your overall values". There are problems here, because people do profess social values that they don't really hold (in some sense), but roughly it is what they would reflect on and come up with.
Someone could value the short-term more than the long-term, but I think that most don't. I'm unsure if this is a side-effect of Christianity-influenced morality or just a strong tendency of human thought.

Locally optimal is probably the correct framing, but that it is irrational relative to whatever idealized values the individual would have. Just like how a hacky approximation of a Chess engine is irrational relative to Stockfish—they both can be roughly considered to have the same goal, just one has various heuristics and short-term thinking that hampers it. These heuristics can be essential, as it runs with less processing power, but in the human mind they can be trained and tuned.

Though I do agree that smoking isn't always irrational: I would say smoking is irrational for the supermajority of human minds, however. The social negativity around smoking may be what influences them primarily, but I'd consider that just another fragment of being irrational— >90% of them would have a value for their health, but they are varying levels of poor at weighting the costs and the social negativity response is easier for the mind to emulate. Especially since they might see people walking around them while they're out taking a cigarette. (Of course, the social approval is some part of a real value too; though people have preferences about which social values they give into)

I think the limitations to radius set by material strength only apply directly to a cylinder spinning by itself without an outer support structure. For example, I think a rotating cylinder habitat surrounded by giant ball bearings connecting it to a non-rotating outer shell can use that outer shell as a foundation, so each part of the cylinder that is "suspended" between two adjacent ball bearings is like a suspension bridge of that length, rather than the whole thing being like a suspension bridge of length equal to the total cylinder diameter. Obviously you would need really smooth, low-friction bearings for this to be a plan to consider, although they would also help with wobble. One way of reducing the friction would be a Russian doll configuration of nested cylinders where each one out was rotating less fast than the previous, which (along with bearings etc) could maybe work. 

On a similar vein, you could replace the mechanical bearings with a gas or fluid, in which the cylinder is immersed.  Similar advantages in damping the wobble modes and (for fluids or very high pressure gases) helping support the cylinder against its own centrifugal weight. The big downside again would be friction.

If you promote "diversity" then you have not only take in mind what you mean with it, but also how policy is likely going to work in practice. 

In practice, there are some dimensions that are easy to measure like race and gender. There are other dimensions that are harder to measure. Some dimensions are also not conducive to research progress. Researchers with IQ under a hundred are underrepresented in grant giving.

Then there are variables like vaccination status, where being unvaxxed does not result in you having a worse ability to do research in the same way as having a lower IQ but there are perspectives on medical research that will correlate with vaccination status. 

If your policy tries to increase the representation of unvaxxed researchers, that might be threatening to hegemonic beliefs and thus a research bureaucracy likely prefers increasing representation of minority races that are unlikely to threaten any hegenomic beliefs. 

If you don't specify the dimensions, the dimensions that are going to selected are most likely those that don't threaten hegemony of current opinions and thus the dimensions that are least likely to actually matter for diversity of ideas and the selected dimensions might even be chosen to strengthen the hegemony of the existing ideas. 

If you actually want real diversity by doing things like calling for diversity in vaccination status you should do that explicitly. 

If this was the setup I would bet on "hard man" fitness people swearing that running with the spin to run in a little more than earth normal gravity was great for building strength and endurance and some doctor somewhere would be warning people that the fad may not be good for your long term health.

Here's a perspective on AI automating everything I haven't seen before, which is relevant to AI governance.

AI being able to automate robotics and AI research will eventually transform the physical world into something which resembles a lot more like the online/virtual world.

Depending on the speed of AI research, this may either happen in several months, or more like a decade, but there's a plausible path to AI turning the physical world into something more like an online/virtual world.

The implications for AI governance are somewhat similar to the implications of companies moderating social media in the present.

Here's several implications:

1. You really can't have the level of liberty and autonomy that people today have around basically everything, for the same reason that there is no actual free speech rights for social media, or really any of the rights we take for granted, and one of the basic reasons for this is that it's very low cost to disrupt online spaces, and even if we avoid the vulnerable world hypothesis that posits tech that is so destructive as to be an x-risk and so widespread that you essentially need to have a totalitarian state to prevent it from being developed without effective defenses, it's likely that there exists some means to constantly troll and degrade the discourse in far more effective ways, and one of the reasons why a lot of social media is as strict about moderation as it is today is because it's too easy to degrade the discourse and just troll everyone in a way that real life doesn't have, ad it's too easy to destroy in social media relative to creation.

2. As a corollary, this means that alignment of governments to citizens become far more important in the future than in the 18th-21st centuries, because we likely have to remove a lot of the checks like democracy that ensure that a misaligned leader doesn't destroy a nation, and suffice it to say that the alignment of social media companies to their users is not going to cut it.

3. One failure mode of social media that we should try to avoid is that the platforms don't really have any ability to hold nuanced conversations for a number of reasons.