A forum for researchers to publicly discuss safety issues in advanced AI

There's nothing wrong with talking about true essences. When I described Yudkowsky's "step backward" into philosophy (and thinking about 'the true essence of intelligence'), I was speaking positively, of a behavior I endorse and want to see more of. My point was that progress toward engineering can exhibit a zigzagging pattern; I think we currently need more zags than zigs, but it's certainly possible that at some future date we'll be more zig-deprived than zag-deprived, and philosophy will get prioritized.

'Formalization' and mathematical logic is closer to what MIRI has in mind when it says 'mathematics'. See http://intelligence.org/research-guide.

But that's been of no use in developing real world AI

It's arguably been useful in building models of AI safety. To quote Exploratory Engineering in AI:

A Monte-Carlo approximation of AIXI can play Pac-Man and other simple games (Veness et al. 2011), but some experts think AIXI approximation isn’t a fruitful path toward human-level AI. Even if that’s true, AIXI is the first model of cross-domain intelligent behavior to be so completely and formally specified that we can use it to make formal arguments about the properties which would obtain in certain classes of hypothetical agents if we could build them today. Moreover, the formality of AIXI-like agents allows researchers to uncover potential safety problems with AI agents of increasingly general capability—problems which could be addressed by additional research, as happened in the field of computer security after Lampson’s article on the confinement problem.

AIXI-like agents model a critical property of future AI systems: that they will need to explore and learn models of the world. This distinguishes AIXI-like agents from current systems that use predefined world models, or learn parameters of predefined world models. Existing verification techniques for autonomous agents (Fisher, Dennis, and Webster 2013) apply only to particular systems, and to avoiding unwanted optima in specific utility functions. In contrast, the problems described below apply to broad classes of agents, such as those that seek to maximize rewards from the environment.

For example, in 2011 Mark Ring and Laurent Orseau analyzed some classes of AIXIlike agents to show that several kinds of advanced agents will maximize their rewards by taking direct control of their input stimuli (Ring and Orseau 2011). To understand what this means, recall the experiments of the 1950s in which rats could push a lever to activate a wire connected to the reward circuitry in their brains. The rats pressed the lever again and again, even to the exclusion of eating. Once the rats were given direct control of the input stimuli to their reward circuitry, they stopped bothering with more indirect ways of stimulating their reward circuitry, such as eating. Some humans also engage in this kind of “wireheading” behavior when they discover that they can directly modify the input stimuli to their brain’s reward circuitry by consuming addictive narcotics. What Ring and Orseau showed was that some classes of artificial agents will wirehead—that is, they will behave like drug addicts.

Fortunately, there may be some ways to avoid the problem. In their 2011 paper, Ring and Orseau showed that some types of agents will resist wireheading. And in 2012, Bill Hibbard (2012) showed that the wireheading problem can also be avoided if three conditions are met: (1) the agent has some foreknowledge of a stochastic environment, (2) the agent uses a utility function instead of a reward function, and (3) we define the agent’s utility function in terms of its internal mental model of the environment. Hibbard’s solution was inspired by thinking about how humans solve the wireheading problem: we can stimulate the reward circuitry in our brains with drugs, yet most of us avoid this temptation because our models of the world tell us that drug addiction will change our motives in ways that are bad according to our current preferences.

Relatedly, Daniel Dewey (2011) showed that in general, AIXI-like agents will locate and modify the parts of their environment that generate their rewards. For example, an agent dependent on rewards from human users will seek to replace those humans with a mechanism that gives rewards more reliably. As a potential solution to this problem, Dewey proposed a new class of agents called value learners, which can be designed to learn and satisfy any initially unknown preferences, so long as the agent’s designers provide it with an idea of what constitutes evidence about those preferences.

Practical AI systems are embedded in physical environments, and some experimental systems employ their environments for storing information. Now AIXI-inspired work is creating theoretical models for dissolving the agent-environment boundary used as a simplifying assumption in reinforcement learning and other models, including the original AIXI formulation (Orseau and Ring 2012b). When agents’ computations must be performed by pieces of the environment, they may be spied on or hacked by other, competing agents. One consequence shown in another paper by Orseau and Ring is that, if the environment can modify the agent’s memory, then in some situations even the simplest stochastic agent can outperform the most intelligent possible deterministic agent (Orseau and Ring 2012a).

I feel as though you're engaging in pedantry for pedantry's sake. The point is that if we can't even solve the simplified version of the problem, there's no way we're going to solve the hard version--effectively, it's saying that you have to crawl before you can walk. Your response was to point out that walking is more useful than crawling, which is really orthogonal to the problem here--the problem being, of course, the fact that we haven't even learned to crawl yet. AIXI and Bayes are useful in that solving AGI problems in the context provided can act as a "stepping stone" to larger and bigger problems. What are you suggesting as an alternative? That MIRI tackle the bigger problems immediately? That's not going to work.

I think you have an idea from our previous discussions why I don't think you physicalism, etc, is relevant to ethics.

Yes some humans seem to have adopted this view where intelligence moves from being a tool and having instrumental value to being instrinsically/terminally valuable. I find often the justifcation for this to be pretty flimsy, though quite a few people seem to have this view. Let's hope a AGI doesn't lol.