Posts
Comments
A case for holding false beliefs
Instrumental rationality: Rationality is about systematized winning.
Epistemic rationality: Rationality is about having true beliefs.
These are correlated. Having true beliefs makes it likelier that you win.
However - IMO there are enough contexts in which having false beliefs (or false pseudo-beliefs) is useful, and perhaps necessary to win.
Organizations, Deceit, Principal-Agent problems
Quite often, if you're trying to achieve a large goal, it's most effective to do this in the context of an organization. This can often be quite a large organization too - one where you can't unilaterally make decisions. I'll center my examples on that case.
- Principle-Agent problems: You're in an organization and want to convince them to take strategy X. You believe strategy Y is likely better for the company, but less good for you. You anticipate strategy X will let your team shine more, and on average give you more power.
Thus, to win, you write strategy X on the bottom line, and then figure out chains of logic that are be maximally convincing to others. Equity EV: You start a startup to enact a specific vision upon the world. Now you must convince investors and employees that they should join you. To do this, you must convince them that the EV of your equity is very high.
This is particularly effective with employees: who are ill-informed and perhaps more willing to buy into a high EV estimate of their equity.Thus, to win, you write valuation X on the bottom line, and then figure out the logic that is maximally convincing to others.
False beliefs make for effective deception
Okay, so you've figured out that you need to deceive others to optimize for your goals. Does that really mean you need false beliefs?
Three routes:
- Avoid this path: There are probably other approaches that need less deception. Go find them, and win by those paths. If big corporation politics needs you to lean into the principal-agent problem, work on something else.
This is a way to live with your values intact. Maybe, if you believe in your values enough, you think this must result in systematized winning too.
However I think that's false. Maximum winning IMO is often easier with occasionally feigning beliefs / soldier mind-setting. - Intentionally hold false beliefs
- Hold true beliefs, and continuously deceive others:
Claim: Holding true beliefs and continuously deceiving others is surprisingly hard.
One reason: you often do need to have "depth" to your beliefs to convince others, ie: have many downstream consequences of the belief precomputed, and you might want to act in accordance with them.
These are smart people, it's hard to deceive them, you're continuously bearing the load of simulating a different belief.
It's not implausible for there to exist false beliefs (eg: strategy X is best for company") where it'd be instrumentally useful for you to believe it, just so that you could convince others better.
Another reason is that deceiving, and believing that you're deceiving is bad. People do not want to deceive, or belief they are deceitful. It's easier to wrap this up into a kinda-fake belief.
Lastly, being caught intentionally deceiving can come with incredibly severe consequences. Thus, you really really want to simulate holding this false belief. You shouldn't drop your mask to anyone.
Can you really knowingly hold false beliefs?
So, you've held a false belief and propagated its implications through your beliefs in great depth. Mostly just because it's instrumentally useful.
Is this really your belief? If you were captured by a malicious genie and had to bet on outcomes predicted by this belief, you would recant.
On the other hand: this seems very belief-y. You actually do spend much more time modeling the world as if this belief is true, as that's most useful for convincing others.
You maybe even generally come to believe that you hold this belief, only realizing you don't when the malicious genie gets you!
I'm happy to call these pseudo-beliefs instead - but I think they explain a lot. I do believe that many winning strategies that interface with people/organizations involve holding these false pseudo-beliefs.
Appendix
- Tweet that triggered this post
- I do still think that truth-seeking is useful on the margin. Likely a losing proposition to heavily hold false beliefs/pseudo-beliefs, but note that this is useful & common.
I believe you often see this in practice via cordoned off areas where truth-seeking is more seriously applied.
- ie: Epistemic rationality as not this overarching primary framework, but merely a tool in your toolbox that must be applied sparingly lest you take down load-bearing false beliefs.
What are all the high-level answers to "What should you, a layperson, do about AI x-risk?". Happy to receive a link to an existing list.
Mine from 5m of recalling answers I've heard:
- Don’t work for OpenAI
- Found or work for an AI lab that gains a lead on capabilities, while remaining relatively safe
- Maybe work for Anthropic, they seem least bad
- Don’t work for any AI lab
- Don’t take any action which increases revenue of any AI lab
- Mourn
- Do technical AI alignment
- Don’t do technical AI alignment
- Do AI governance & advocacy
- Donate to AI x-risk funds
- Cope
- Don't perform domestic terrorism
Labor magnification as a measure of AI systems.
Cursor is Mag(Google SWE, 1.03) if Google would rather have access to Cursor than 3% more SWEs at median talent level.
A Mag(OpenAI, 10) system is a system that OpenAI would rather have than 10x more employees at median talent level
A time based alternative is useful too, in cases where it's a little hard to envision as many more employees.
A tMag(OpenAI, 100) system is a system that OpenAI would rather have than 100x time-acceleration for its current employee pool.
Given that definition, some notes:
1. Mag(OpenAI, 100) is my preferred watermark for self-improving AI, one where we'd expect takeoff unless there's a sudden & exceedingly hard scaling wall.
2. I prefer this framework over t-AGI in some contexts - as I expect that we'll see AI systems able to do plenty of 1-month tasks before it can do all 1-minute tasks. OpenAI's Deep Research already comes close to confirming this. Magnification is more helpful as it measures shortcomings based on whether they can be cheaply shored up by human labor.