Reactive devaluation: Bias in Evaluating AGI X-Risks

if the proof is to be formal, it needs to plug into the mathematical formalizations one would use to do the social science form of this. and if you want to deal with certain kinds of social systems, it's more complicated than that. I'm also an experienced generalist nerd with a lot to say; unfortunately, even if we're real good at babbling, they're right to look at people funny even if they have the systems programming experience or what have you. seems like you're doing stuff that doesn't plug into reality very well. can you provide a demonstration of it as a multi-agent simulation, for example? various levels of shareable multi-agent sim tool to demonstrate whatever effect you'd like; then the only proof you need provide circumstantially is the applicability of the simulation, which means if we find a way the metaphor breaks, we can go back to the drawing board for the simulated example; in software and life in general, tests only give us coverage of the example we try, but at least we can assert that the test passes.

• https://hash.ai/platform/engine - https://hash.ai/models?sort=popularity (looks really sleek and modern, can do complicated cellular automata or smooth movement stuff)
• https://insightmaker.com/explore (designed for sharing and has existing simulations of ... varying quality)
• https://simlin.com/
• https://www.collimator.ai/ <- fancy
• https://ncase.me/loopy/ <- cheesy demo one
• https://github.com/evoplex/evoplex <- cellular automata focused
• https://neuralmmo.github.io/ <- neural network training focused, designed for "love in a simbox" style usage, from openai. there are a few more from openai and deepmind I skipped mentioning, browse their githubs if curious.
• honorable mention, looks kinda stuffy: https://helipad.dev/
• this one has built in causal inference tools, which is cool I guess? https://github.com/zykls/whynot

The problem of a very poor signal to noise ratio from messages received from people outside of the established professional group basically means that the risk of discarding a good proposal from anyone regarded as an outsider is especially likely.

This insight feels relevant to a comment exchange I was in yesterday [LW(p) · GW(p)]. An AI Safety insider (Christiano) lightly read an overview of work by an outsider (Landry).  The insider then judged the work to be "crankery", in effect acting as a protecting barrier against other insiders having to consider the new ideas.

The sticking point was the claim "It is 100% possible to know that X is 100% impossible", where X is a perpetual motion machine or a 'perpetual general benefit machine' (ie. long-term safe and beneficial AGI).

The insider believed this was an exaggerated claim, which meant we first needed to clarify epistemics and social heuristics, rather than the substantive argument form. The reactions by the busy "expert" insider, who had elected to judge the formal reasoning, led to us losing trust that they would proceed in a patient and discerning manner.

There was simply not enough common background and shared conceptual language for the insider to accurately interpret the outsider's writings ("very poor signal to noise ratio from messages received").  

Add to that:

• Bandwagon effect
  • "the tendency to believe that [long-term safe AGI is possible] because many other people do"
• Naive realism 
  • "that the facts are plain for all to see; that rational people will agree with us [that long-term safe AGI is possible]; and that those who do not are either uninformed, lazy, irrational, or biased."
• Belief bias 
  • "Where the evaluation of the logical strength of an argument is biased by the believability of the conclusion [that long-term safe AGI is impossible]... The difficulty is that we want to apply our intuition too often, particularly because it is generally much faster/easier than actually doing/implementing analytic work.)...  Arguments which produce results contrary to one's own intuition about what "should" or "is expected" be the case are also implicitly viewed as somewhat disabling and invalidating of one's own expertise, particularly if there also is some self-identification as an 'expert'. No one wants to give up cherished notions regarding themselves. The net effect is that arguments perceived as 'challenging' will be challenged (criticized) somewhat more fully and aggressively than rationality and the methods of science would have already called for."
• Conservatism bias 
  • "People do not want to be seen as having strong or 'extreme opinions', as this in itself becomes a signal from that person to the group that they are very likely to become 'not a member' due to their willingness to prefer the holding of an idea as a higher value than they would prefer being regarded as a member in good standing in the group. Extreme opinions [such as that it is 100% possible to know that long-term safe AGI is 100% impossible] are therefore to be regarded as a marker of 'possible fanaticism' and therefore of that person being in the 'out crowd'."
•  Status quo bias;  System justification
  • "The tendency to like things to stay relatively the same. The tendency to defend and bolster the status quo [such as resolving to build long-term safe AGI, believing that it is a hard but solvable problem]. Existing social, economic, and political arrangements tend to be preferred, and alternatives disparaged sometimes even at the expense of individual and collective self-interest."
• Reactance
  • "The degree to which these various bias effects occur is generally in proportion to a motivating force, typically whenever there is significant money, power, or prestige involved. Naturally, doing what someone 'tells you to do' [like accepting the advice to not cut to the chase and instead spend the time to dig into and clarify the arguments with us, given the inferential distance] is a signal of 'low status' and is therefore to be avoided whenever possible, even if it is a good idea."

I mean, someone recognised as an expert in AI Safety could consciously mean well trying to judge an outsider's work accurately – in the time they have. But that's a lot of biases to counteract.


Forrest actually clarified the claim further to me by message:

Re "100%" or "fully knowable":  

By this, I usually mean that the analytic part of an argument is fully finite and discrete, and that all parts (statements) are there, the transforms are enumerated, known to be correct etc (ie, is valid). 

In regards to the soundness aspect, that there is some sort of "finality" or "completeness" in the definitions, such that I do not expect that they would ever need to be revised (ie, is at once addressing all necessary aspects, sufficiently, and comprehensively), and that the observations are fully structured by the definitions, etc. Usually this only works for fairly low level concepts, things that track fairly closely to the theory of epistemology itself -- ie, matters of physics that involve symmetry or continuity directly (comparison) or are expressed purely in terms of causation, etc. 

One good way to test the overall notion is that something is "fully 100% knowable" if one can convert it to a computer program, and the program compiles and works correctly. The deterministic logic of computers cannot be fooled, as people sometimes can, as there is no bias. This is may be regarded by some as a somewhat high standard, but it makes sense of me as it is of the appropriate type: Ie, a discrete finite result being tested in a purely discrete finite environment. Hence, nothing missing can hide.

But the point is – few readers will seriously consider this message.  

That's my experience, sadly. 
The common reaction I noticed too from talking with others in AI Safety is that they immediately devaluated that extreme-sounding conclusion that is based on the research of an outsider.  A conclusion that goes against their prior beliefs, and against their role in the community.