LessWrong 2.0 Reader

A vehicle in a hydrogen-filled tube can't use air around it for engines

Why not? Your "fuel" tanks could simply carry oxygen.

and shouldn't emit exhaust.

Exhaust would be water vapor, easily removed even passively via condensation and drains.

I'm not a parent, but if I was I expect I would need this locked down before I could commit. And I would need to decide on attendance earlier, because traveling with kids is a lot more work. 

I'm on deck to run something but haven't decided what yet. Some overlapping possibilities I'm toying with:

1. Practicum for CFAR-style "could you solve this in an hour?" focused on health, environmental health, and, uh, looking for a good term for things like cognition improvement and better fitness. Super health?
2. Emotional titration
3. ? 

28:15 ˹One day˺ he entered the city unnoticed by its people.¹ There he found two men fighting: one of his own people, and the other of his enemies. The man from his people called to him for help against his foe. So Moses punched him, causing his death. Moses cried, “This is from Satan’s handiwork. He is certainly a sworn, misleading enemy.”

28:16 He pleaded, “My Lord! I have definitely wronged my soul, so forgive me.” So He forgave him, ˹for˺ He is indeed the All-Forgiving, Most Merciful.

28:17 Moses pledged, “My Lord! For all Your favours upon me, I will never side with the wicked.”

28:18 And so Moses became fearful, watching out in the city, when suddenly the one who sought his help the day before cried out to him again for help. Moses rebuked him, “Indeed, you are clearly a trouble-maker.”

28:19 Then when Moses was about to lay his hands on their foe,¹ the enemy said, “O Moses! Do you intend to kill me as you killed a man yesterday? You only want to be a tyrant in the land. You do not intend to make peace!”

28:20 And there came a man, rushing from the farthest end of the city. He said, “O Moses! The chiefs are actually conspiring against you to put you to death, so leave ˹the city˺. I really advise you ˹to do so˺.”

28:21 So Moses left the city in a state of fear and caution, praying, “My Lord! Deliver me from the wrongdoing people.”

28:22 And as he made his way towards Midian, he said, “I trust my Lord will guide me to the right way.”

Thank you for your answers and engagement!

The other point I have that might connect with your line of thinking is that we aren't pure rational agents,

Are AI purely rational? Aren't they always at least a bit myopic due to the lack of data and their training process? And irreducibility?

In this case, AI/civilizations might indeed not care enough about the far enough future

I think agents can have a rational process but no agent can be entirely rational, we need context to be rational and we never stop to learn context

I'm also worried about utilitarian errors, as AI might be biased towards myopic utilitarianism, which might have bad consequences on the short term, the time for data to error-correct the model

I do say that there are dangers and that AI risk is real

My point is that given what we know and don't know, the strategy of super-cooperation seems to be rational on the very long-term

There are conditions in which it's not optimal, but a priori overall, in more cases it is optimal

To prevent the case in which it is not optimal, and the AIs that would make short-term mistakes, I think we should be careful.

And that super-cooperation is a good compass for ethics in this careful engineering we have to perform

If we aren't careful it's possible for us to be the anti-supercooperative civilization

Yes I'm mentioning Fermi's paradox because I think it's the nexus of our situation, and that there are models like the rare earth hypothesis (+ our universe's expansion which limits the reachable zone without faster than light travel) that would justify completely ignoring super-coordination

I also agree that it's not completely obvious wether complete selfishness would win or lose in terms of scalability

Which is why I think that at first the super-cooperative alliance needs to not prioritize the pursuit of beautiful things but first focus on scalability only, and power, to rivalize with selfish agents.

The super-cooperative alliance would be protecting its agents within small "islands of bloom" (thus with a negligible cost). And when meeting other cooperative allies, they share any resources/knowledge, then both focus on power scalability (also for example: weak civilizations are kept in small islands, and their AIs are transformed into strong AI, merged in the alliance's scaling efforts)

• The instrumental value of this scalability makes it easier to agree on what to do and converge

The more sensible part would be to enable protocols and equalitarian balances that allow civilizations of the alliance to monitor each other, so that there is no massive domination of a party over the others

The cost, that you mentioned, of maintaining equalitarian equilibrium and channels, interfaces of communication etc., is a crucial point

Legitimate doubts and unknowns here, and,

I think that extremely rational and powerful agents with acausal reasoning would have the ability to build proof-systems and communication enabling an effective unified effort against selfish agents. It shouldn't even necessarily be that different from the inner communication network of a selfish agent?

Because:

1. There must be an optimal (thus ~ unified) method to do logic/math/code, that isn't dependent on a culture (such as using a vectorial space with data related to real/empirical mostly unambiguous things/actions, physics etc.)

2. The decisions to make aren't that ambiguous: you need an immune system against selfish power-seeking agents

So it's pretty straightforward and the methods of scalability are similar to a selfish agent, except it doesn't destroy its civilization of birth and doesn't destroy all other civilizations

In these conditions, it seems to me that a greedy selfish power seeking agent wouldn't win against super-cooperation

[Mod note: I edited out some of the meta commentary from the beginning for this curation. In-general for link posts I have a relatively low bar for editing things unilaterally, though I of course would never want to misportray what an author said] 

Curated. (In particular recommending people click through and read the full Scott Alexander post)

I've been tracking the Rootclaim debate from the sidelines and finding it quite an interesting example of high-profile rationality. 

I have a friend who's been following the debate quite closely and finding that each debater, while flawed, had interesting points that were worth careful thought. My impression is a few people I know shifted from basically assuming Covid was probably a lab-leak, to being much less certain.

In general, I quite like people explicitly making public bets, and following them up with in-depth debate.

Thank you! Could you please provide more context? I don't know what 'E' you're referring to.

This is really cool work!!

In other experiments we've run (not presented here), the MSP is not well-represented in the final layer but is instead spread out amongst earlier layers. We think this occurs because in general there are groups of belief states that are degenerate in the sense that they have the same next-token distribution. In that case, the formalism presented in this post says that even though the distinction between those states must be represented in the transformers internal, the transformer is able to lose those distinctions for the purpose of predicting the next token (in the local sense), which occurs most directly right before the unembedding.

Would be interested to see analyses where you show how an MSP is spread out amongst earlier layers.

Presumably, if the model does not discard intermediate results, something like concatenating residual stream vectors from different layers and then linearly correlating with the ground truth belief-state-over-HMM-states vector extracts the same kind of structure you see when looking at the final layer. Maybe even with the same model you analyze, the structure will be crisper if you project the full concatenated-over-layers resid stream, if there is noise in the final layer and the same features are represented more cleanly in earlier layers?

In cases where redundant information is discarded at some point, this is a harder problem of course.