it does not have to turn into a paperclip maximizer, but could turn into a billion other things that are equally bad and behaves unexpectedly. Again, to me unexpected means that it has behave in a way that no one had foreseen or predicted. Hence that any guardrails would work would be just by chance or luck. Either we know how the system will behave and we can control that, or we don't. Any unexpected or surprising behavior COULD spell disaster. If you that think that this kind of system behavior is to be expected you are in my world just saying that you expect disaster to happen.