LessWrong 2.0 Reader

I just made an anki deck for this, using gpt 4o-mini to label images of faces. I’ll try to report back in a month or two how well it works.

My current best guess: Subsidiarity I've been thinking along these lines for the past few years, but I feel like my thinking was clarified and boosted by Allison's recent series: Gaming the Future [LW · GW]

The gist of the idea is to create clever systems of decentralized control and voluntary interaction which can still manage to coordinate on difficult risky tasks (such as enforcing defensive laws against weapons of mass destruction). Such systems could shift humanity out of the Pareto suboptimal lose-lose traps and races we are stuck in. Win-win solutions to our biggest current problems seem possible, and coordination seems like the biggest blocker.

I am hopeful that one of the things we can do with just-before-the-brink AI will be to accelerate the design and deployment of such voluntary coordination contracts.

Alignment isn’t like that; it was chosen to be an important problem

Like medicine.

This was specifically commented on in a book whose preface I read as a child.  It was called something like "Medicine: from science to magic", and I have not found a clear link back to it.

No, you're right that aristocracy is more complicated. There were lots of pressures that shaped the form of it. Certainly more than how good of managers aristocrats made!

An invalid syllogism: "The rules of aristocracy were shaped by forces. Avoiding poor management is a force. Therefore, the rules of aristocracy will be all about avoiding poor management."

Aristocrats were also selected for how well they could extract rents from those below, and how well they could resist rent-extraction from above, both alone and collectively. Nor was the top-down pressure all about making aristocrats into productive managers - rent-extraction has been mentioned, and also weakening the aristocracy to secure central power, allowing advancement via marriage and alliance, various human status games, and the need for a legislative arm of government.

I don't want to hear the One Pressure That Explains Everything (but only qualitatively, and if you squint). I'll want to hear when they have the dozen pressures that make up a model that can be quantitatively fit to past data by tuning some parameters, including good retrodictive accuracy over a held-out time period.

Over the past few days I've been doing a lit review of the different types of attention heads people have found and/or the metrics one can use to detect the presence of those types of heads. 

Here is a rough list from my notes, sorry for the poor formatting, but I did say its rough!

• Bigram entropy
• positional embedding ablation
• prev token attention
• prefix token attention
• ICL score
• comp scores
• multigram analysis
• duplicate token score
• induction head score
• succession score
• copy surpression heads
• long vs short prefix induction head [LW · GW] differentiation
• induction head specializations
  • literal copying head
  • translation
  • pattern matching
• copying score
• anti-induction heads
• S-inhibition heads
• Name mover heads
• Negative name mover heads
• Backup name mover heads
• (I don't entirely trust this paper) Letter mover heads
• (possibly too specific to be useful) year identification heads
  • also MLPs which id which years are greater than the selected year
• (I don't entirely trust this paper) queried rule locating head
• (I don't entirely trust this paper) queried rule mover head
• (I don't entirely trust this paper) "fact processing" head
• (I don't entirely trust this paper) "decision" head
• (possibly too specific) subject heads
• (possibly too specific) relation heads
• (possibly too specific) mixed subject and relation heads

That the argument that ASI could easily (relative to the sun's own efforts) dismantle the star completely was barely argued for, and his priors weren't moved much and he wanted object-level reasons to believe it was feasible to rapidly dismantle the sun.

What do you think Richard Ngo claimed about this?

Seems like we've now established that we largely agree on the explicit propositions we've stated all through this thread. Given that, your initial response feels to me like a bit of a non-sequitur.

As I understand it, your response argued against a universal claim that social pressure always inverts genuine preferences, while I had explicitly made the narrower claim that this sometimes happens and is worth watching out for. Does that seem like a fair characterization? If so, can you help me understand why your initial response felt important and relevant to you in context?

I agree with Richard Ngo and Simon that any dismantling of the sun is going to be a long-term project, and this matters.

What is the precise statement for being able to use surreal numbers when we remove the Archimedean axiom? The surreal version of the VNM representation theorem in "Surreal Decisions" (https://arxiv.org/abs/2111.00862) seems to still have a surreal version of the Archimedean axiom.

Re the parent example, I was imagining that the 2-priority utility function for the parent only applied after they already had children, and that their utility function before having children is able to trade off between not having children, having some who live, and having some who die. Anecdotally it seems a lot of new parents experience diachronic inconsistency in their preferences.