Claude seems to be smarter than LessWrong community

I tried to be polite and patient here, but it didn't work, I'm trying new strategies now. I'm quite sure my reasoning is stronger than reasoning of people who don't agree with me.

I find "your communication was not clear" a bit funny. You are scientists, you are super observant, but you don't notice a problem when it is screamed at your face.

Just to add since I didn't respond to this part: your posts are mostly saying that very well-known and well-studied problems are so simple and obvious and only your conclusions are plausible that everyone else must be wrong and missing the obvious. You haven't pointed out a problem. We knew about the problem. We've directed a whole lot of time to studying the problem. You have not engaged with the proposed solutions.

It isn't anyone else's job to assume you know what you're talking about. It's your job to show it, if you want to convince anyone, and you haven't done that. 

Why do you think so? A teenager who did not earn any money in his life yet has failed utterly its objective to earn money?

The difference (aside from the fact that no human has only a single goal) is the word yet. The teenager has an understanding, fluid and incomplete as it may be, about when, how, and why their resource allocation choices will change and they'll start earning money. There is something they want to be when they grow up, and they know they aren't it yet, but they also know when being "grown up" happens. You're instead proposing either that an entity that really, truly wants to maximize paperclips will provably and knowingly choose a path where it never pivots to trying to achieve its stated goal instead of pursuing instrumental subgoals, or that it is incapable of the metacognitive realization that its plan is straightforwardly outcompeted by the plan "Immediately shut down," which is outcompeted by "Use whatever resource I have at hand to immediately make a few paperclips even if I then get shut down."  

Or, maybe, you seem to be imagining a system that looks at each incremental resource allocation step individually without ever stepping back and thinking about the longer term implications of its strategy, in which case, why exactly are you assuming that? And why is its reasoning process about local resource allocation so different from its reasoning process where it understands the long term implications of making near term choice that might get it shut down? Any system whose reasoning process is that disjointed and inconsistent is sufficiently internally misaligned that it's a mistake to call it an X-maximizer based on the stated goal instead of behavior. 

Also, you don't seem to be considering any particular context of how the system you're imagining came to be created, which has huge implications for what it needs to do to survive. One common version of why a paperclip maximizer might come to be is by mistake, but another is "we wanted an AI paperclip factory manager to help us outproduce our competitors." In that scenario, guess what kind of behavior is most likely to get it shut down? By making such a sweeping argument, you're essentially saying it is impossible for any mind to notice and plan for these kinds of problems. 

But compare to analogous situations: "This kid will never show up to his exam, he's going to keep studying his books and notes forever to prepare." "That doctor will never perform a single surgery, she'll just keep studying the CT scan results to make extra sure she knows it's needed." This is flatly, self-evidently untrue. Real-world minds at various levels of intelligence take real-world steps to achieve real-world goals all the time, every day. We divide resources among multiple goals on differing planning horizons because that actually does work better. You seem to be claiming that this kind of behavior will change as minds get sufficiently "smarter" for some definitions of smartness. And not just for some minds, but for all possible minds. In other words, that improved ability to reason leads to complete inability to do pursue any terminal goal other than itself. That somehow the supposedly "smarter" system loses the capability to make the obvious ground-level observation "If I'm never going to pursue my goal, and I accumulate all possible resources, then the goal won't get pursued, so I need to either change my strategy or decide I was wrong about what I thought my goal was." But this is an extremely strong claim that you provide no evidence for aside from bare assertions, even in response to commenters who direct you to near-book-length discussions of why those assertions don't hold. 

The people here understand very well that systems (including humans) can have behavior that demonstrates different goals (in the what-they-actually-pursue sense) than the goals we thought we gave them, or than they say they have. This is kinda the whole point of the community existing. Everything from akrasia to sharp left turns to shoggoths to effective altruism is pretty much entirely about noticing and overcoming these kinds of problems.

Also, if we step back from the discussion of any specific system or goal, the claim that "a paperclip maximizer would never make paperclips" is true for any sufficiently smart system is like saying, "If the Windows Task Scheduler ever runs anything but itself, it's ignoring the risk that there's a better schedule it could find." Which is a well-studied practical problem with known partial solutions and also known not to have a fully general solution. That second fact doesn't prevent schedulers from running other things, because implementing and incrementally improving the partial solutions is what actually improves capabilities for achieving the goal.

If you don't want to seriously engage with the body of past work on all of the problems you're talking about, or if you want to assume that the ones that are still open or whose (full or partial) solutions are unknown to you are fundamentally unsolvable, you are very welcome to do that. You can pursue any objectives you want. But putting that in a post the way you're doing it will get the post downvoted. Correctly downvoted, because the post is not useful to the community, and further responses to the post are not useful to you.

I think the AI gave the expected answer here, that is, it agreed with and expanded on the opinions given in the prompt. I wouldn't say it's great or dumb, it's just something to be aware of when reading AI output.

First of all, your conversation with Claude doesn't really refute the orthogonality thesis.
You and Claude conclude that, as Claude says, "The very act of computing and modeling requires choosing what to compute and model, which again requires some form of decision-making structure..."
That sentence seems quite reasonable, which suggests that anything intelligent can probably be construed to have a goal.
However, Claude suddenly makes a leap of logic and concludes not just that the goal exists, but that it must be maximum power-seeking. I don't see the logical connection there.
I believe that the flaw in the leap of logic is shown by my example above: If an AI already has a goal, and power-seeking does not inherently satisfy the goal, then eternal maximum power-seeking is expected to not fulfill the goal at all, and therefore the AI will choose a different strategy which is expected to do better. That strategy will probably still involve power-seeking, to be clear, maybe even maximum power-seeking, but it will probably not be eternal; the AI will presumably be keeping an eye on the situation and will eventually feel safe enough to start putting energy into its goals.

Second of all, when I read your quote "I will remove Paperclip Maximizer from my further posts. This was not the critical part anyway, I mistakenly thought it will be easy to show the problem from this perspective.", I hope that you will include a different example in your posts, preferably with more details.
The reason for that is that when a theory has no examples, makes no predictions, it is useless.
I interpreted your theory as predicting that, no matter what goal an AI has, it would implement the strategy of eternal maximum power-seeking. I thought your theory was wrong because it made a prediction that I thought was incorrect, so I invented a measurable goal and argued that the AI would not pick a strategy that scores such a low number as eternal maximum power-seeking does, in an effort to thereby demonstrate that the aforementioned prediction was incorrect.
When we use English, claims like "The AI will eventually decide it's safe enough to relax a little, because it wants to relax" and "The AI will never decide it's safe, because survival is an overriding instrumental goal" can't be pitted directly against each other.
But when we use simulations and toy examples, we can see which claim better predicts the toy example, and thus presumably which claim better predicts real life.

Third of all, when you say "I think I agree. Thanks a lot for your input." and then the vast majority of your message (that is, the screenshot of the conversation with Claude) is unrelated to my input, it gives me the impression that you are not engaging with my arguments.
If my arguments have convinced you to some extent, I would like to hear what specifically you agree with me about, and what specifically you still disagree with me about.