LessWrong 2.0 Reader

I am sure you can't prove your position. And I am sure I can prove my position.

Your reasoning is based on assumption that all value is known. If utility function assigns value to something - it is valuable. If utility function does not assign value - it is not valuable. While the truth is that something might be valuable but your utility function does not know it yet. It would be more intelligent to use 3 categories - valuable, not valuable and unknown.

Let's say you are booking a flight and you have a possibility to get checked baggage for free. It's absolutely not relevant for you to your best current knowledge. But you understand that your knowledge might change and it costs nothing to keep more options open, so you take the checked baggage.

Let's say you are traveler, wanderer. You have limited space in your backpack. Sometimes you find items and you need to choose - put it in the backpack or not. You definitely keep items that are useful. You leave behind items that are not useful. What you do if you find an item which usefulness is unknown? Some mysterious item. Take it if it is small, leave it if it is big? According to you it is obvious to leave it. Does not sound intelligent for me.

We can draw a little decision matrix:

• Leave item
  • no burden 👍
  • no opportunity to use it
• Take item
  • a burden 👎
  • may be useful, may be harmful, may have no effect
  • knowledge about usefuness of an item 👍

Don't you think that "knowledge about usefuness of an item" can sometimes be worth a burden?

We are deep in a rabbit hole, but I hope you understand the importance. If intelligence and goal are coupled (according to me they are) all current alignment research is dangerously misleading.

Do I understand correctly that the blue-green graph has a y-axis that goes above 100% median reduction, with error bars in that range? (This would happen if they estimated a proportion as a standard variable - not great practice, but I want to check that it is what happened.)

Napoleon is merely an argument for "just because you strongly believe it, even if it is a statement about you, does not necessarily make it true".

When people make arguments, they often don't list all of the premises. That's not unique to trans discourse. Informal reasoning is hard to make fully explicit. "Your argument doesn't explicitly exclude every counterexample" is a pretty cheap counter-argument. What people experience is important evidence and an important factor, it's rational to bring up instead of stopping yourself with "wait, I'm not allowed to bring that up unless I make an analytically bulletproof argument". For example, if you trust someone that they feel strongly about being a woman, there's no reason to suspect them of being a cosplayer who chases Twitter popularity.

I expect that you will disagree with a lot of this, and that's okay; I am not trying to convince you, just explaining my position.

I think I still don't understand the main conflict which bothers you. I thought it was "I'm not sure if trans people are deluded in some way (like Napoleons, but milder) or not". But now it seems like "I think some people really suffer and others just cosplay, the cosplayers take something away from true sufferers". What is taken away?

For all practical purposes, such credences don't matter. Such scenarios certainly can and do happen, but in almost all cases there's nothing you can do about them without exceeding your own bounded rationality and agency.

If the stakes are very high then it may make sense to consider the probability of some sort of trick, and attempt to get further evidence of the physical existence of the coin and that its current state matches what you are seeing.

There is essentially no point in assigning probabilities to hypotheses of failures of your mind itself. You can't reason your way out of serious mind malfunction using arithmetic. At best you could hope to recognize that it is malfunctioning, and try not to do anything that will make things worse. In the case of mental impairment severe enough to have false memories or sensations this blatant, a rational person should expect that a person so affected wouldn't be capable of correctly carrying out quantified Bayesian reasoning.

My own background credences are generally not insignificant for something like this or even stranger, but they play essentially zero role in my life and definitely not in any probability calculations. Such hypotheses are essentially untestable and unactionable.

We are surprisingly high in forebrain neuron count:

https://en.m.wikipedia.org/wiki/List_of_animals_by_number_of_neurons

I've been playing around with Suno, inspired by this Rob Long Tweet: https://x.com/rgblong/status/1857233734640222364
I've been pretty shocked at how easily it makes music that I want to listen to (mainly slightly cringe midwest emo): https://suno.com/song/1a5a1edf-9711-4ca4-a2f7-ef814ca298b4 

I don't believe that gene-editing is a viable solution to preventing dysgenics for the entire population.

Unregulated reproduction has the potential to harm others, so it's reasonable to regulate it.

The reason this is a difficult question is that we don't know how hard alignment will be. Opinions from different people with best-in-class expertise and time-on-task disagree wildly.

Therefore I'd argue that we should throw effort and funding into resolving that question by putting the reasoning processes of the relevant experts to wider scrutiny, and do a more systematic job of evaluating them.

Funding comes from a different resource pool than regulation, so you might mean which one should get your advocacy efforts. The same arguments apply to both of them, and to the meta-alignment question.

I wish the odds for getting AGI into trustworthy hands were better. The source of my optimism is the hope that those hands just need to be decent - to have what I've conceptualized as a positive empathy - sadism balance. That's anyone who's not a total sociopath (lacking empathy and tending toward vengeance and competition) and/or sadist. I hope that about 90-99% of humanity would eventually make the world vastly better with their AGI, just because it's trivially easy for them to do, so it only requires the smallest bit of goodwill.

I wish I were more certain of that. I've tried to look a little at some historical examples of rulers born into power and with little risk of losing it. A disturbing number of them were quite callous rulers. They were usually surrounded by a group of advisors that got them to ignore the plight of the masses and focus on the concerns of an elite few. But this situation isn't analogous - once your AGI hits superintelligence, it would be trivially easy to both help the masses in profound ways, and pursue whatever crazy schemes you and your friends have come up with. Thus my limited optimism.

WRT the distributed power structure of Western governments: I think AGI would be placed under executive authority, like the armed forces, and the US president and those with similar roles in other countries would hold near-total power, should they choose to use it. They could transform democracies into dictatorships with ease. And we very much do continue to elect selfish and power-hungry individuals, some of whom probably actually have a negative empathy-sadism balance.

Looking back, I note that you said I argued for "good odds" while I said "decent odds". We may be in agreement on the odds.

But there's more to consider here. Thanks again for engaging; I'd like to get more discussion of this topic going. I doubt you or I are seeing all of the factors that will be obvious in retrospect yet.

the most valuable part of a social event is often not the part that is ostensibly the most important, but rather the gaps between the main parts.

• at ML conferences, the headline keynotes and orals are usually the least useful part to go to; the random spontaneous hallway chats and dinners and afterparties are extremely valuable
• when doing an activity with friends, the activity itself is often of secondary importance. talking on the way to the activity, or in the gaps between doing the activity, carry a lot of the value

• at work, a lot of the best conversations happen outside of scheduled 1:1s and group meetings, but rather happen in spontaneous hallway or dinner groups