A rejection of the Orthogonality Thesis

1. There is a large mind design space. Do we have any actual reasons for thinking so? Sure, one can argue everything has a large design space, but in practice, there's often an underlying unique mechanism for how things work.

I don't see how this relates to the Orthogonality Thesis. For a given value or goal, there may be many different cognitive mechanisms for figuring out how to accomplish it, or there may be few, or there may be only one unique mechanism. Different cognitive mechanisms (if they exist [LW · GW]) might lead to the same or different conclusions about how to accomplish a particular goal. 

For some goals, such as re-arranging all atoms in the universe in a particular pattern, it may be that there is only one effective way of accomplishing such a goal, so whether different cognitive mechanisms are  able to find the strategy for accomplishing such a goal is mainly a question of how effective those cognitive mechanisms are. The Orthogonality Thesis is saying, in part, that figuring out how to do something is independent of wanting to do something, and that the space of possible goals and values is large. If I were smarter, I probably could figure out how to tile the universe with tiny squiggles, but I don't want to do that, so I wouldn't.

2. Ethics are not an emergent property of intelligence - but again, that's just an assertion. There's no reason to believe or disbelieve it. It's possible that self-reflection (and hence ethics and the ability to question one's goals and motivations) is a pre-requisite for general cognition - we don't know whether this is true or not because we don't really understand intelligence yet.

I don't see what ability to self-reflect has to do with ethics. It's probably true that anything superintelligent is capable, in some sense, of self-reflection, but why would that be a problem for the Orthogonality Thesis? Do you believe that an agent which terminally values tiny molecular squiggles would "question its goals and motivations" and conclude that creating squiggles is somehow "unethical"? If so, maybe review the metaethics [? · GW] sequence; you may be confused about what we mean around here when we talk about ethics, morality, and human values.

The previous two are assertions that could be true, but reflective stability is definitely not true - it's paradoxical.

I think reflective stability, as it is usually used on LW, means something more narrow than how you're interpreting it, and is not paradoxical. It's usually used to describe a property of an agent following a particular decision theory [? · GW]. For example, a causal decision theory agent is not reflectively stable, because on reflection, it will regret not having pre-committed in certain situations [? · GW]. Logical decision theories are more reflectively stable in the sense that their adherents do not need to pre-commit to anything, and will therefore not regret not making any pre-commitments when reflecting on their own minds and decision processes, and how they would behave in hypothetical or future situations.

So if you reject the Orthogonality Thesis, what map between capability and goals are you using instead?

This line of reasoning is absurd: it assumes an agent knows in advance the precise effects of self-improvement — but that’s not how learning works! If you knew exactly how an alteration in your understanding of the world would impact you, you wouldn’t need the alteration: to be able to make that judgement, you’d have to be able to reason as though you had already undergone it.

It seems there is some major confusion is going on here - it is, generally speaking, imporrible to know the outcome of an arbitrary computation without actually running it, but that does not mean it's impossible to design a specific computation in a way you'd know exactly what the effects would be. For example, one does not need to know the trillionth digit of pi in order to write a program that they could be very certain would compute that digit.

You also seem to be too focused on minor modifications of a human-like mind, but focusing too narrowly on minds is also missing the point - focus on optimization programs instead.

For many different kinds of X, it should be possible to write a program that given a particular robotics apparatus (just the electromechanical parts without a specific control algorithm), predicts which electrical signals sent to robot's actuators would result in more X. You can then place that program inside the robot and have the program's output wired to the robot controls. The resulting robot does not "like" X, it's just robotically optimizing for X.

The orthogonality principle just says that there is nothing particularly special about human-aligned Xs that would make the X-robot more likely to work well for those Xs over Xs that result in human extinction (e.g. due to convergent instrumental goals, X does not need to specifically be anti-human).

So if the argument the OT proponents are making is that AI will not self-improve out of fear of jeopardising its commitment to its original goal, then the entire OT is moot, because AI will never risk self-improving at all.

This seems to me to apply only to self improvement that modifies the outcome of decision-making irrespective of time. How does this account for self improvement that only serves to make decision making more efficient? 

If I have some highly inefficient code that finds the sum of two integers by first breaking them up into 10000 smaller decimal values, randomly orders them and then adds them up in serial, and I rewrite the code to do the same thing but in way less ops, I have self improved without jeopardizing my goal.

This kind of self improvement can still be fatal in the context of deceptively aligned systems.

I can't upvote this sadly, because I do not have the karma, but I would if I did. 

There is another post [LW · GW] about this as well. 

Don't be too taken aback if you receive negative karma or some pushback - it is unfortunately expected for posts on this topic taking a position anti to the Orthogonality Thesis.