Why Stop AI is barricading OpenAI

I agree that with superficial observations, I can't conclusively demonstrate that something is devoid of intellectual value.

Thanks for recognising this, and for taking some time now to consider the argument. 

However, the nonstandard use of words like "proof" is a strong negative signal on someone's work.

Yes, this made us move away from using the term “proof”, and instead write “formal reasoning”. 

Most proofs nowadays are done using mathematical notation. So it is understandable that when people read “proof”,  they automatically think “mathematical proof”. 

Having said that, there are plenty of examples of proofs done in formal analytic notation that is not mathematical notation. See eg. formal verification practices in the software and hardware industries, or various branches of analytical philosophy.

If someone wants to demonstrate a scientific fact, the burden of proof is on them to communicate this in some clear and standard way

Yes, much of the effort has been to translate argument parts in terms more standard for the alignment community.

What we cannot expect is that the formal reasoning is conceptually familiar and low-inferential distance. That would actually be surprising – why then has someone inside the community not already derived the result in the last 20 years?

The reasoning is going to be as complicated as it has to be to reason things through. 

This problem is exacerbated when someone bases their work on original philosophy. To understand Forrest Landry's work to his satisfaction someone will have to understand his 517-page book An Immanent Metaphysics

Cool that you took a look at his work. Forrest’s use of terms is meant to approximate everyday use of those terms, but the underlying philosophy is notoriously complicated. 

Jim Rutt is an ex-chair of Santa Fe Institute who defaults to being skeptical of metaphysics proposals (funny quote he repeats: “when someone mentions metaphysics, I reach for my pistol”).  But Jim ended up reading Forrest’s book and it passed his B.S. detector. So he invited Forrest over to his podcast for a three-part interview. Even if you listen to that though, I don’t expect you to immediately come away understanding the conceptual relations.

So here is a problem that you and I are both seeing:

• There is this polymath who is clearly smart and recognised for some of his intellectual contributions (by interviewers like Rutt, or co-authors like Anders).
• But what this polymath claims to be using as the most fundamental basis for his analysis would take too much time to work through.
• So then if this polymath claims to have derived a proof by contradiction –concluding that long-term AGI safety is not possible – then it is intractable for alignment researchers to verify the reasoning using his formal annotation and his conceptual framework. That would be asking for too much – if he’d have insisted on that, I agree that would have been a big red flag signalling crankery.
• The obvious move then is for some people to work with the polymath to translate his reasoning to a basis of analysis that alignment researchers agree is a sound basis to reason from. And to translate to terms/concepts people are familiar with. Also, the chain of reasoning should not be so long that busy researchers never end up reading through, but also not so short that you either end up having to use abstractions readers are unfamiliar with, or open up unaddressed gaps in the reasoning.  Etc.
• The problem becomes finding people who are both willing and available to do that work. One person is probably not enough. 

Having read the research proposal my guess is that they will prove something roughly like the Good Regulator Theorem or Rice's theorem

Both are useful theorems, which have specific conclusions that demonstrate that there are at least some limits to control. 

(ie. Good Regulator Theorem demonstrates a limit to a system’s capacity to model – or internally functionally represent – the statespace of some more complex super-system. Rice Theorem demonstrates a particular limit to having some general algorithm predict a behavioural property of other algorithms.)

The hashiness model is a tool meant for demonstrating under conservative assumptions – eg. of how far from cryptographically hashy the algorithm run through ‘AGI’ is, and how targetable human-safe ecosystem conditions are – that AGI would be uncontainable. With “uncontainable”, I mean that no available control system connected with/in AGI could constrain the possibility space of AGI’s output sequences enough over time such that the (cascading) environmental effects do not lethally disrupt the bodily functioning of humans.

Paul expressed appropriate uncertainty. What is he supposed to...say...?

I can see Paul tried expressing uncertainty by adding “probably” to his claim of how the entire scientific community (not sure what this means) would interpret that one essay.

To me, it seemed his commentary was missing some meta-uncertainty. Something like “I just did some light reading. Based on how it’s stated in this essay, I feel confident it makes no sense for me to engage further with the argument. However, maybe other researchers would find it valuable to spend more time engaging with the argument after going through this essay or some other presentation of the argument.” 

~
That covers your comments re: communicating the argument in a form that can be verified by the community. 

Let me cook dinner, and then respond to your last two comments to dig into the argument itself. EDIT: am writing now, will respond tomorrow. 

An obvious consideration for any reflective agent is to find ways to reduce the risk of goal-related failure.

…

by "goal-related systems" I just meant whatever is keeping track of the outcomes being optimized for.

So the argument for 3. is that just by AGI continuing to operate and maintain its components as adapted to a changing environment, the machinery can accidentally end up causing destabilising effects that were untracked or otherwise insufficiently corrected for. 

You could call this a failure of the AGI’s goal-related systems if you mean with that that the machinery failed to control its external effects in line with internally represented goals. 

But this would be a problem with the control process itself.

An obvious consideration for any reflective agent is to find ways to reduce the risk of goal-related failure.

Unfortunately, there are fundamental limits to that cap the extent to which the machinery can improve its own control process. 

Any of the machinery’s external downstream effects that its internal control process cannot track (ie. detect, model, simulate, and identify as a “goal-related failure”), that process cannot correct for.  

For further explanation, please see links under point 4.

Decentralizing away from a single point of failure is another obvious step that one would take in a post-ASI world.

The problem here is that (a) we are talking about not just a complicated machine product but self-modifying machinery and (b) at the scale this machinery would be operating at it cannot account for most of the potential human-lethal failures that could result. 

For (a), notice how easily feedback processes can become unsimulatable for such unfixed open-ended architectures. 

• E.g. How can AGI code predict how its future code learned from unknown inputs will function in processing subsequent unknown inputs? What if future inputs are changing as a result of effects propagated across the larger environment from previous AGI outputs? And those outputs were changing as a result of previous new code that was processing signals in connection with other code running across the machinery? And so on.  

For (b), engineering decentralised redundancy can help especially at the microscale. 

• E.g. correcting for bit errors.
• But what does it mean to correct for failures at the level of local software (bugs, viruses, etc)? What does it mean to correct for failures across some decentralised server network? What does it mean to correct for failures at the level of an entire machine ecosystem (which AGI effectively becomes)?

~

In scaling up the connected components, this exponentially increases their degrees of freedom of interaction. And as those components change in feedback with surrounding contexts of the environment (and have to in order for AGI to autonomously adapt), an increasing portion of the possible human-lethal failures cannot be adequately controlled for by the system itself.