Justifiable Erroneous Scientific Pessimism

"Continental drift" is usually the go-to example. For one, the mechanism originally proposed was complete nonsense...

There was a pretty solid basis for believing that 2-dimensional crystals were thermodynamically unstable and thus couldn't exist. Then in 2004 Geim and Novoselov did it (isolated graphene for the first time) and people had to re-scrutinize the theory, since it was obviously wrong somehow. It turns out that the previous theory was correct for 2D crystals of essentially infinite size, but it seems to not apply for non-infinite crystals. At least that is how it was explained to me once by a theorist on the subject.

The opening paragraph of this paper cites the relevant literature: http://cdn.intechopen.com/pdfs/40438/InTech-The_cherenkov_effect_in_graphene_like_structures.pdf

Lord Kelvin was wrong but was he pessimistic? He wasn't saying we could never know the answer, or visit the sun, or anything like that. Yes, he guessed wrongly, and too low, but it doesn't seem to be the case that 'underestimating a quantity' is pessimism. If nothing else, the quantity might be 'number of babies killed'.

The claim that the Sun revolves around the Earth. If the Earth revolved around the Sun, there would have been a parallax in the observations of stars from different positions in the orbit. There was no observable parallax, so Earth probably didn't revolve around the Sun.

Off the top of my head, how about the Landau Pole? A famous and usually right genius calculated that the gauge theories of quantum fields are a dead end, and set the Soviet and to some degree Western physics a few years back, if I recall correctly. His calculation was not wrong, he simply missed the alternate possibilities.

EDIT: hmm, I'm having trouble locating any links discussing the negative effects of the Landau pole discovery on the QED research.

We also know many famous examples of scientists just completely making up their pessimism, for example about the impossibility of human heavier-than-air flight.

This isn't what you asked for, but I might as well enumerate a few of these examples, for everyone's benefit. For the field of AI research:

"You can build a machine to draw [logical] conclusions for you, but I think you can never build a machine that will draw [probabilistic] inferences."

George Pólya (1954), ch. 15 — a few decades before the probabilistic revolution in AI.

[Machines] cannot play chess any more than they can play football.

Mortimer Taube (1960) — not long before computers began to regularly dominate amateur and then expert chess players. (Edit: this one seems wrong)

"[Pattern recognition] is obviously an inductive process, hence it is not a logical process or mechanical sorting that the computer can perform without human aid."

Satosi Watanabe (1974) — a couple decades before both supervised and unsupervised machine learning took off.

Also, Hubert Dreyfus mocked the capabilities of chess computers, and compared AI to alchemy, in Dreyfus (1965) — a mere two years before he was defeated by the chess computer Mac Hack.

Here is another famous example:Chandrasekhar's limit. Eddington rejected the idea of black holes ("I think there should be a law of Nature to prevent a star from behaving in this absurd way!"). Says wikipedia:

Chandra's discovery might well have transformed and accelerated developments in both physics and astrophysics in the 1930s. Instead, Eddington's heavy-handed intervention lent weighty support to the conservative community astrophysicists, who steadfastly refused even to consider the idea that stars might collapse to nothing.

I guess this is not quite what you are asking for, since the math was on Chandrasekhar's side, and Eddington was pinning his hopes on "new physics". To be fair, recent discussions about horizon firewalls) could be such new physics.

The general success rate of breakthroughs is pretty damn low, and so I'd argue that most examples of "invalid" pessimism (excluding some stupid ones coming from scientists you never heard of before coming across a quote, and excluding things like PR campaigning by Edison), viewed in the context of almost all breakthroughs failing for some reason you can't anticipate, are not irrational but simply reflect absence of strong evidence in favour of success (and absence of strong evidence against unknown obstacles), at the time of assessment (and corresponding regression towards the mean rate of success). They're merely not as hindsight resistant as Fermi's example. You look back at history seeing things that succeeded. Go read archive of some old journals, and note the zillions of amazing breakthroughs that did not pan out.

If bomb did not rely on unusual U235 , Fermi would not have been irrational about 10% probability to emission of secondary neutrons from fission - it is something that most likely either happens for all fissions, or does not happen for any fissions, so the clever "there would be one" argument doesn't work irrespective of U235. U235 is not the most general valid objection, it's just the objection for which sources are easiest to find. No one did the silly task of writing out that production of secondary neutrons is not a statistically independent fact across different nuclei, and we're lucky that there's just 1 nucleus so we don't have to, either.

I'm not sure if this is justifiable or just an old-fashioned blunder...

On the subject of stars, all investigations which are not ultimately reducible to simple visual observations are…necessarily denied to us… We shall never be able by any means to study their chemical composition.

-- August Comte, 1835

I'm leaning towards "blunder" myself...

it has been plausibly argued to me that all the roads to nuclear weapons, including plutonium production from U-238, may have bottlenecked through the presence of significant amounts of Earthly U235

This has interesting repercussions for Fermi's paradox.

I posted the following in a quotes page a few months back. I don't know how justifiable these were, and these are only questionably pessimism, but there may be some interesting examples in this. In particular, my light knowledge of the subject suggests that there really were extremely compelling reasons to disregard Feynman's formulation of QED for many years after it was first introduced.

It is interesting to note that Bohr was an outspoken critic of Einstein's light quantum (prior to 1924), that he mercilessly denounced Schrodinger's equation, discouraged Dirac's work on the relativist electron theory (telling him, incorrectly, that Klein and Gordon had already succeeded), opposed Pauli's introduction of the neutrino, ridiculed Yukawa's theory of the meson, and disparaged Feynman's approach to quantum electrodynamics.

[Footnote to: "This was a most disturbing result. Niels Bohr (not for the first time) was ready to abandon the law of conservation of energy". The disturbing result refers to the observations of electron energies in beta-decay prior to hypothesizing the existence of neutrinos.]

-David Griffiths, Introduction to Elementary Particles, 2008 page 24

Here's an example of the 'opposite' - a case of unjustifiable correct optimism:

Columbus knew the Earth was round but should also have known the radius of the Earth and size of Eurasia well enough to know that the voyage East to Asia was simply impossible with the ships and supplies he went with. It seems to have turned out OK for him, though.

This is probably not a very useful example and I wouldn't be surprised to see that there were plenty more of these examples.

Kuhn's Structure of Scientific Revolutions is all about how an old scientific approach is often more right than the new school -- fits the data better, at least in the areas widely acknowledged to be central. Only later does the new approach become refined enough to fit the data better.

Thomas Malthus' view that in the long run we will always be stuck in (what we now call) the Malthusian trap. He would have been right if not for the sustained growth given to us by the industrial revolution.

This blog post claims that only a few years before the Wright brother's success, the consensus was that flying machines would necessarily have to be less dense than air (like hot air balloons).

it has been plausibly argued to me that all the roads to nuclear weapons, including plutonium production from U-238, may have bottlenecked through the presence of significant amounts of Earthly U235 (apparently even the giant heap of unrefined uranium bricks in Chicago Pile 1 was, functionally, empty space with a scattering of U235 dust).

All is such a strong word unless supplemented with qualifiers. I question the plausibility the arguments at supporting that absolute. The route "wait for an extra century or two of particle physics research and spend a few trillion producing the initial seed stock" would still be available.