LessWrong 2.0 Reader

Thirty million dollars is a lot of money, but there are plenty of smart rich people who don't mind taking risks. So, once the identity and (apparent) motives of the Trump whale were revealed, why didn't a handful of them mop up the free EV?

Well, first I think you're right to say "a handful". My (limited but nonzero) experience of "sufficiently rich" people who made their money in "normal" ways, as opposed to by speculating on crypto or whatever, is that they're too busy to invest a lot of time in playing this kind of market personally, especially if they have to pay enough attention to play it intelligently. They're not very likely to employ anybody else to play for them either. Many or most of them will see as the whole thing as basically an arcane, maybe somewhat disreputable game. So the available pool is likely smaller than you might think.

That conjecture is at least to some degree supported by the fact that nobody, or not enough people, stepped in when the whole thing started. Nothing prevented the market from moving so far to begin with. It may not have been as certain what was going on then, but things looked weird enough that you'd expect a fair number of people to decide that crazy money was likely at work, and step in to try to take some of it... if enough such people were actually available.

In any case, whether when the whole thing started, after public understanding was reasonably complete, or anywhere along the way, the way I think you'd like to make your profit on the market being miscalibrated would be to buy in, wait for the correction, and then sell out... before the question resolved and before unrelated new information came in to move the price in some other way.

But would be hard to do that. All this is happening potentially very close to resolution time, or at least to functional resolution time. The market is obviously thin enough that single traders can move it, and new information is coming in all the time, and the already-priced-in old information isn't very strong and therefore can't be expected to "hold" the price very solidly, and you have to worry about who may be competing with you to take the same value, and you may be questioning how rational traders in general are^[1].

So you can't be sure you'll get your correction in time to sell out; you have a really good chance of being stuck holding hold your position through resolution. If "markets can remain irrational longer than you can remain solvent", then they can also stay irrational for long enough that trading becomes moot.

If you have to hold through resolution, then you do still believe you have positive expected value, but it's really uncertain expected value. After all, you believe the underlying question is 50-50, even if one of those 50s would pay you more than the other would lose you. And you have at best limited chance to hedge. So you have to have risk tolerance high enough that, for most people, it'd be in the "recreational gambling" range rather than the "uncertain investment" range. The amount of money that any given (sane) person wants to put at risk definitely goes down under that much uncertainty, and probably goes down a lot. So you start to need more than a "handful" of people.

Also, don't forget the point somebody made the other day about taxes. Unless you're a regular who plays many different questions in such volume that you expect to offset your winnings with losses, you're going to stand to "win" a double-digit percentage less than you stand to "lose", whether on selling off your position or on collecting after resolution. Correcting 60-40 to 50-50 may just plain not be profitable even if you collect.

There are probably other sources of friction, too.

Sorry this was confusing! From our definition here:

We’ll use “pragmatic principles” to refer to principles according to which belief-forming or decision-making procedures should “perform well” in some sense.

• "Avoiding dominated strategies" is pragmatic because it directly evaluates a decision procedure or set of beliefs based on its performance. (People do sometimes apply pragmatic principles like this one directly to beliefs, see e.g. this work on anthropics.)
• Deference isn't pragmatic, because the appropriateness of your beliefs is evaluated by how your beliefs relate to the person you're deferring to. Someone could say, "You should defer because this tends to lead to good consequences," but then they're not applying deference directly as a principle — the underlying principle is "doing what's worked in the past."

That's true in any scenario, though. For all of life really. The ratio of likelihood of loopholes costing you money versus losing the best costing you money doesn't matter, as long as their absolute values are both low. And shoring up ambiguous language in the bet is how you make the former low. 

I also partook in some betting last time, but I'd like to do more. I've done more research on this over the last year and built more conviction that this is the right play, that's really all that's changed. Would you be down to bet again?

Put another way: this post seems like it’s arguing with someone but I’m not sure who.

I'm confused by what you mean by "non-pragmatic". For example, what makes "avoiding dominated strategies" pragmatic but "deference" non-pragmatic? 

(It seems like the pragmatic ones help you decide what to do and the non-pragmatic ones help you decide what to believe, but then this doesn't answer how to make good decisions.)

I think I care a bunch about the subject matter of this post, but something about the way this post is written leaves me feeling confused and ungrounded.

Before reading this post, my background beliefs were:

1. Rationality doesn't (quite) equal Systemized Winning. Or, rather, that focusing on this seems to lead people astray more than helps them.
2. There's probably some laws of cognition to be discovered, about what sort of cognition will have various good properties, in idealized situations.
3. There's probably some messier laws of cognition that apply to humans (but those laws are maybe more complicated).
4. Neither sets of laws necessarily have a simple unifying framework that accomplishes All the Things (although I think the search for simplicity/elegance/all-inclusiveness is probably a productive search, i.e. it tends to yield good stuff along the way. "More elegance" is usually achievable on the margin.
5. There might be heuristics that work moderately well for humans much of the time, which approximate those laws.
   1. there are probably Very Rough heuristics you can tell an average person without lots of dependencies, and somewhat better heuristics you can give to people who are willing to learn lots of subskills.

Given all that... is there anything in-particular I am meant to take from this post? (I have right now only skimmed it, it felt effortful to comb for the novel bits). I can't tell whether the few concrete bits are particularly important, or just illustrative examples.

I'd like to see research that uses Jonathan Haidt's Moral Foundations research and the AI risk repository to forecast whether there will soon be a moral backlash / panic against frontier AI models.

Ideas below from Claude:

# Conservative Moral Concerns About Frontier AI Through Haidt's Moral Foundations

## Authority/Respect Foundation
### Immediate Concerns
- AI systems challenging traditional hierarchies of expertise and authority
- Undermining of traditional gatekeepers in media, education, and professional fields
- Potential for AI to make decisions traditionally reserved for human authority figures

### Examples
- AI writing systems replacing human editors and teachers
- AI medical diagnosis systems challenging doctor authority
- Language models providing alternative interpretations of religious texts

## Sanctity/Purity Foundation
### Immediate Concerns
- AI "thinking" about sacred topics or generating religious content
- Artificial beings engaging with spirituality and consciousness
- Degradation of "natural" human processes and relationships

### Examples
- AI spiritual advisors or prayer companions
- Synthetic media creating "fake" human experiences
- AI-generated art mixing sacred and profane elements

## Loyalty/Ingroup Foundation
### Immediate Concerns
- AI systems potentially eroding national sovereignty
- Weakening of traditional community bonds
- Displacement of local customs and practices

### Examples
- Global AI systems superseding national control
- AI replacing local business relationships
- Automated systems reducing human-to-human community interactions

## Fairness/Reciprocity Foundation
### Immediate Concerns
- Unequal access to AI capabilities creating new social divides
- AI systems making decisions without clear accountability
- Displacement of workers without reciprocal benefits

### Examples
- Elite access to powerful AI tools
- Automated systems making hiring/firing decisions
- AI wealth concentration without clear social benefit

## Harm/Care Foundation
### Immediate Concerns
- Potential for AI systems to cause unintended societal damage
- Psychological impact on children and vulnerable populations
- Loss of human agency and autonomy

### Examples
- AI manipulation of emotions and behavior
- Impact on childhood development
- Dependency on AI systems for critical decisions

## Key Trigger Points for Conservative Moral Panic
1. Religious Content Generation
  - AI systems creating or interpreting religious texts
  - Automated spiritual guidance
  - Virtual religious experiences

2. Traditional Value Systems
  - AI challenging traditional moral authorities
  - Automated ethical decision-making
  - Replacement of human judgment in moral domains

3. Community and Family
  - AI impact on child-rearing and education
  - Disruption of traditional social structures
  - Replacement of human relationships

4. National Identity
  - Foreign AI influence on domestic affairs
  - Cultural homogenization through AI
  - Loss of local control over technology

## Potential Flashpoints for Immediate Backlash
1. Education System Integration
  - AI teachers and tutors
  - Automated grading and assessment
  - Curriculum generation

2. Religious Institution Interaction
  - AI pastoral care
  - Religious text interpretation
  - Spiritual counseling

3. Media and Information Control
  - AI content moderation
  - Synthetic media generation
  - News creation and curation

4. Family and Child Protection
  - AI childcare applications
  - Educational AI exposure
  - Family privacy concerns

This is not very practically useful to me but dayumn it is cool

I do pretty much agree. All laws and international agreements are ultimately enforced by the use of force if need be, so that's not saying anything new. It probably does need to be a hard ban on open-source AI at some point, but that's well in the future, and I think the discussion will look very different once we have clearly parahuman AGI.

This is all going to be a tough pill to swallow. I think it's going to be almost necessary that any government that enacts these rules will also have to assure everyone, and then follow through at least decently well with spreading the benefits of real AGI as broadly as possible. I see some hope in that becoming a necessity. We might get some oversight boards that could at least think clearly and apply some influence toward sanity.