Our work here is not arguing that probing is a perfect solution in general; it's just a single datapoint of how it fares on the models from our Sleeper Agents paper [LW · GW].

What I'm saying is that the people you mention should put a little more time into it. When I've been involved in philosophy discussions with academics, people tend to treat it like a fun game, with the goal being more to sore points and come up with clever new arguments than to converge on the truth.

I think most of the world doesn't take philosophy seriously, and they should.

I think the world thinks "there aren't real answers to philosophical questions, just personal preferences and a confusing mess of opinions". I think that's mostly wrong; LW does tend to cause convergence on a lot of issues for a lot of people. That might be groupthink, but I held almost identical philosophical views before engaging with LW - because I took the questions seriously and was truth-seeking.

I think Musk or Page are fully capable of LW-style philosophy if they put a little time into it - and took it seriously (were truth-seeking).

What would change people's attitudes? Well, I'm hoping that facing serious questions in how we create, use, and treat AI does cause at least some people to take the associated philosophical questions seriously.

Yes. But I think you have mixed up expected value and expected utility. Please show your calculations.

Great post, very clearly written. Going to share it in my spaces.

I do not understand your reasoning. Please show your calculations.

"Stephanie Murray reports that the village thing can still be done, and in particular has pulled off a ‘baby swapping’ system that periodically pools child care so parents can have time for themselves."

Maybe there is more detail in the linked blog but just from this post it sounds like a reinvention of Kindergarten.

Ah, shoot. You're right. Probably not good to use "odds" and "probability" interchangeably for percentages like I did. Should be fixed now.

Yeah that makes sense; the knowledge should still be there, just need to re-shift the distribution "back"

Noting, this doesn't really engage with any of the particular other claims in the previous comment's link, just makes a general assertion. 

The North Wind, the Sun, and Abadar

One day, the North Wind and the Sun argued about which of them was the strongest. Abadar, the god of commerce and civilization, stopped to observe their dispute. “Why don’t we settle this fairly?” he suggested. “Let us see who can compel that traveler on the road below to remove his cloak.”

The North Wind agreed, and with a mighty gust, he began his effort. The man, feeling the bitter chill, clutched his cloak tightly around him and even pulled it over his head to protect himself from the relentless wind. After a time, the North Wind gave up, frustrated.

Then the Sun tried his turn. Beaming warmly from the heavens, the Sun caused the air to grow pleasant and balmy. The man, feeling the growing heat, loosened his cloak and eventually took it off in the heat, resting under the shade of a tree. The Sun began to declare victory, but as soon as he turned away, the man put on the cloak again.

The god of commerce then approached the traveler, jingling a pouch of gold coins in his hand.

“Good sir,” Abadar called out, “that cloak of yours—how much would you sell it for?”

The man considered the offer, his eyes lighting up. “Five coins,” he replied hesitantly.

“Done,” said Abadar, handing over the coins and taking the cloak. The traveler tucked the money away and continued on his way, unbothered by either wind or heat. He soon bought a new cloak and invested the remainder in an index fund. The returns were steady, and in time he prospered far beyond the value of his simple cloak.

“See,” Abadar declared to the Wind and the Sun, “strength lies neither in force nor persuasion but in creating opportunities for mutual benefit. The cloak is mine permanently, and the man is better off as well. My solution also has minimal deadweight loss, assuming the elasticity—”

Before Abadar could say any more, the North Wind grumbled, the Sun conceded, and Abadar strode away, his wisdom proven. Thus, it was decided that commerce, when conducted wisely, can accomplish what neither force nor gentle persuasion alone can achieve.