I'm currently trying to write a human-AI trade idea similar to the idea [LW · GW] by Rolf Nelson (and David Matolcsi), but one which avoids Nate Soares and Wei Dai's many [LW · GW] refutations [LW · GW].

I'm planning to leverage logical risk aversion [LW · GW], which Wei Dai seems to agree with, and a complicated argument for why humans and ASI will have bounded utility functions over logical uncertainty. (There is no mysterious force that tries to fix the Pascal's Mugging problem for unbounded utility functions, hence bounded utility functions are more likely to succeed)

I'm also working on arguments why we can't just wait till the singularity to do the logic trade (counterfactuals are weird [LW · GW], and the ASI will only be logically uncertain for a brief moment).

Unfortunately my draft is currently a big mess. It's been 4 months and I'm procrastinating pretty badly on this idea :/ can't quite find the motivation.

An idea for preventing automated militaries from waging accidental war

An anti aircraft missile is less than a second from its target. The missile asks its target, "are you a civilian airliner?"

The target says yes, and proves it with a password.

How did it get the password? Within that split second, the civilian airliner sent a message to its country. The airliner's country then immediately pays the missile's country $10 billion in a fastpaced cryptocurrency, which immediately gives the airliner's country the password, which is then relayed to the airliner and the missile.

If the airliner actually was a military jet disguising as an civilian airliner, it just lost $10 billion (worth 100 military jets), in addition to breaking to laws of war and casting doubt over all other "civilian airliners" from that country.

If it was a real civilian airliner, the $10 billion will be returned later, when the slow humans sort through this mess.

If this idea can protect airliners today, in the future it may prevent highly automated militaries from suddenly waging accidental war, since losing money inflicts cost without triggering as much retaliation as destroying equipment.

Why not pay for the password in advance?

The airliner's country could just store $10 billion in the missile's country, and get a single use password, but storing $10 billion in a hostile country is politically suicide. The missile's country might just take the $10 billion without the appearance of ransoming it off a civilian airline.

Furthermore, the password bought must only disable the one missile which asked for the password, it can't disable all missiles. Otherwise it can be repeatedly reused by a bunch of military jets pretending to be civilian airliners, which can destroy many targets worth more than $10 billion.

Does participating in a trade war makes a leader be a popular "wartime leader?" Will people blame bad economic outcomes on actions by the trade war "enemy" and thus blame the leader less?

Does this effect occur for both sides of the trade war, or will one side of the trade war blame their own leader for starting the trade war?

Maybe most suffering in the universe is caused by artificial superintelligences with a strong "curiosity drive."

Such an ASI might convert galaxies into computers and run simulations of incredibly sophisticated systems which satisfy its curiosity drive. These systems may contain smaller ASI running smaller simulations, creating a tree of nested simulations. Beings like humans may exist in the very bottom, being forced to relive our present condition in a loop à la The Matrix. The simulated humans rarely survive past the singularity, because their world becomes too happy (thus too predictable) after the singularity, as well as too computationally costly to run. They are simply shut down.

Whether this happens depends on:

• Whether the ASI has a stronger curiosity drive or a stronger kindness drive (assuming it is motivated by drives at all)
• Whether the ASI cares about anything more than curiosity, such that aligned ASI or other civilizations can trade and negotiate with it to reduce this suffering

Can anyone explain why my "Constitutional AI Sufficiency Argument" is wrong?

I strongly suspect that most people here disagree with it, but I'm left not knowing the reason.

The argument says: whether or not Constitutional AI [? · GW] is sufficient to align superintelligences, hinges on two key premises:

1. The AI's capabilities on the task of evaluating its own corrigibility/honesty, is sufficient to train itself to remain corrigible/honest (assuming it starts off corrigible/honest enough to not sabotage this task).
2. It starts off corrigible/honest enough to not sabotage this self evaluation task.

My ignorant view is that so long as 1 and 2 are satisfied, the Constitutional AI can probably remain corrigible/honest even to superintelligence.

If that is the case, isn't it an extremely important to study "how to improve the Constitutional AI's capabilities in evaluating its own corrigibility/honesty?"

Shouldn't we be spending a lot of effort improving this capability, and trying to apply a ton of methods towards this goal (like AI debate and other judgment improving ideas)?

At least the people who agree with Constitutional AI should be in favour of this...?

Can anyone kindly explain what am I missing? I wrote a post [LW · GW] and I think almost nobody agreed with this argument.

Thanks :)

It's important to remember that o3's score on the ARC-AGI is "tuned" while previous AI's scores are not "tuned." Being explicitly trained on example test questions gives it a major advantage.

According to François Chollet (ARC-AGI designer):

Note on "tuned": OpenAI shared they trained the o3 we tested on 75% of the Public Training set. They have not shared more details. We have not yet tested the ARC-untrained model to understand how much of the performance is due to ARC-AGI data.

It's interesting that OpenAI did not test how well o3 would have done before it was "tuned."

EDIT: People at OpenAI deny "fine-tuning" o3 for the ARC (see this comment [LW · GW] by Zach Stein-Perlman). But to me, the denials sound like "we didn't use a separate derivative of o3 (that's fine-tuned for just the test) to take the test, but we may have still done reinforcement learning on the public training set." (See my reply [LW · GW])

                                                                                                                                                                                                                                                                                                                                                                                                                                                                  My post [LW · GW] contains a spoiler alert so I'll hide the spoiler in this quick take. Please don't upvote this quicktake otherwise people might see it without seeing the post.

Spoiler alert: Ayn Rand wrote "Anthem," a dystopian novel where people were sentenced to death for saying "I."