Why pessimism sounds smart

Optimism is what agency looks like to people without it.

Why is this style of pessimism repeatedly wrong?

I liked this post, and upvoted, but this sounds like it might be somewhat cherry-picking? If you say that the pessimism is "repeatedly wrong", then you are ignoring the areas where it has been repeatedly right. E.g. perpetual motion machines and faster-than-light travel continue to remain impossible, as does curing aging (even though we might now begin to be in a position to actually cure it, people have tried various means of becoming immortal for probably thousands of years, and so far the pessimism has always been right). 

In the limit, the pessimists will eventually be correct. There's only so many technologies and improvements that can be made under the laws of physics (e.g. Landauer's limit), and once we hit those real limits, that'll be as good as it gets.

Of course, the catch that a lot of people might not realize is that a) we're at least centuries away from hitting said limits, even with aligned superintelligence, and b) those limits are far, far better than what we have and live with today.

It's not as pithy, but it seems likely that it's better said that some optimists make money, no?

Of course, this doesn't have any direct bearing on this interesting post.

Why is this style of pessimism repeatedly wrong? How can this optimism be justified?

Pessimism is very often right, and optimism wrong (especially about expected timeframes).  The missing piece is the payoff - pessimism can be right most of the time, and optimism wrong most of the time, but most of the progress comes from the somewhat rare cases that deviate from median projections.  And the optimists, while mostly wrong, drive toward those payoffs more effectively than pessimists.

I am reminded of the Python (Monty) parable of the flying sheep: https://www.ibras.dk/montypython/episode02.htm . there is no hope of success, but they don't get rid of Harold (the ringleader) because of the "enormous commercial possibilities should he succeed".

No known solutions can solve our hardest problems—that’s why they’re the hardest ones.

I like the energy, but I have to register a note of dissent here.

Quite a few of our hardest problems do have known solutions - it's just that those known solutions are, or appear, too hard to implement.

• Brute force algorithms exist for almost everything we care about, up to and including AGI.
• Overweight individuals know that if they eat less, they will eventually lose weight; it's just often frustratingly beyond them, for one reason or another. Same with alcoholics and drink.
• Collective action problems have been studied in economics for decades, and by this point a lot of them have clever approaches we could use to at least partway ameliorate them.

'Appear' is important here. Ken Thompson's advice to "when in doubt use brute force" is very good advice because so much resistance can and will crumble to sustained effort. But this requires a somewhat different kind of optimism to the optimism I think you're describing of inventing a fundamentally new solution - it's the optimism to get back up after you fall down and run straight at the wall again until it does.

Building on this, there's also the evolutionary element. Humans continued evolving after civilization, even if the timeframes were too short for much mutation to occur. During early civilizations, and probably most civilizations before the industrial revolution, pessimistic outlooks were extremely common in the real world. As a result, both instinctive and learned pessimism would give an individual more surviving offspring within any given civilization.

Just bouncing something of a reaction to the idea that pessimism sounds smart(er than optimism).

I always hear that humans are, generally, risk-averse. That can be a "smart" strategy, perhaps from some evolutionary imperatives. So we're naturally incline to over-weight possible harms while under-weighting the possible benefits.

But, in that setup pessimism sounds smart because we want to believe we have more to fear on the downside so is perhaps a form of confirmation bias at work.

Why is this style of pessimism repeatedly wrong?

Beware selection bias. If it wasn't repeatedly wrong, there's a good chance we wouldn't be here to ask the question!

The opposite view is that progress is a matter of luck.

Hm. I tend to not view the pessimistic side as luck so much as 'there's a finite number of useful techs, which we are rapidly burning through'.

Extreme, apocalyptic pessimism has an equal and opposite problem....you need to persuade people that they are going to be killed by a novel threat, or by something known that has never killed everybody before.