In the Sequences where Eliezer Yudkowsky presents his view on philosophy and rationality, he explicitly says that he eventually found this sort of old writing to be severely flawed and that much of his philosophy is based on avoiding these errors. Here is a link to the middle of the most relevant sequence, but some of the surrounding writings may be necessary to fully understand the points: https://www.lesswrong.com/s/SXurf2mWFw8LX2mkG/p/Yicjw6wSSaPdb83w9 [? · GW]

Almost sure current Eliezer disagrees with this kind of reasoning. Generally, with this kind of attempt to derive something from nothing, by the sheer power of "but certainly, a sufficiently smart AI would see the possibility of arriving to the conclusion X, and therefore would conclude X" where X just happens to be what the author believes.

The AI would follow the reasoning outlined in the article only if it is specifically programmed to follow exactly that kind of reasoning... in which case, it is not completely fair to say it does not have any pre-established goals.

EDIT:

More importantly, there is a difference between a system having a goal, and a system using a token labeled "goal". You can write a Python program that will output the string "it is meaningful to create art". That does not mean that if you run that program, it will actually start creating art. Similarly, an AI programmed to generate statements might, following the steps outlined in the linked article, derive a sequence of tokens that say that something is meaningful. That doesn't mean that the AI would actually try to do something about it.

To me the very notion of an AI system not having any goals at all seems inherently wrong. If the system is doing something - even if that something is just reasoning - the system must have some means of deciding what to do out of the infinite pool of things that could possibly be done. Whatever that system is defines the goal.

Goal-directed behaviour can be as simple as what a central heating thermostate does. An AI could very possibly have no internal representation of what its own goal is, but if it is carrying out computations, it almost certainly has something which directs it on what sort of computations it's expected to carry out, and that is quite enough to define a goal for it.

There's a recent paper by a human philosopher, which proposes a counterpart of Pascal's wager, but regarding the possible existence of objective good and evil, rather than the possible existence of God. (The uncertain starting point is called metaethical agnosticism. Ethical agnosticism is where you don't know what's right and what's wrong; metaethical agnosticism is uncertainty about the nature of right and wrong.) 

That paper's proposal is basically that you shouldn't do things which might be bad, just in case they actually are bad. 

Eliezer's thought process in 2000 starts in the same place, but goes a little differently, because he has the cognitive optimism that a superintelligence might be able to dispel the uncertainty, and actually know for sure, whether or not there are objective rights and wrongs. 

So this is undoubtedly a possible thought process. But I'd say it's far from an inevitable one. It requires a "mind" that has the concept of goal, that is capable of conceiving of objectively valuable or compelling goals, that is capable of being motivationally affected by that possibility - and which doesn't think up some other clever argument that is even more compelling than this one. 

In principle, sure, why not. In practice, not with any AI system I've seen; training data is needed, and the training algorithm must include an objective in order to generate intelligence at all.