Reflections on My Own Missing Mood

If these paths are viable, I desire to believe that they are viable.

If these paths are nonviable, I desire to believe that they are nonviable.

Does it do any good, to take well-meaning optimistic suggestions seriously, if they will in fact clearly not work? Obviously, if they will work, by all means we should discover that, because knowing which of those paths, if any, is the most likely to work is galactically important. But I don't think they've been dismissed just because people thought the optimists needed to be taken down a peg. Reality does not owe us a reason for optimism.

Generally when people are optimistic about one of those paths, it is not because they've given it deep thought and think that this is a viable approach, it is because they are not aware of the voluminous debate and reasons to believe that it will not work, at all. And inasmuch as they insist on that path in the face of these arguments, it is often because they are lacking in security mindset - they are "looking for ways that things could work out", without considering how plausible or actionable each step on that path would actually be. If that's the mode they're in, then I don't see how encouraging their optimism will help the problem.

Is the argument that any effort spent on any of those paths is worthwhile compared to thinking that nothing can be done?

edit: Of course, misplaced pessimism is just as disastrous. And on rereading, was that your argument? Sorry if I reacted to something you didn't say. If that's the take, I agree fully. If one of those approaches is in fact viable, misplaced pessimism is just as destructive. I just think that the crux there is whether or not it is, in fact, viable - and how to discover that.

Just a side note that climate change would not lead to the biosphere collapse even in the worst case. Earth is pretty cold and low on life now compared to 55 million years ago, when it was hotter than the worst projections of the climate models. If anything, global warming will be good for the biosphere, though maybe not so much for humans.

Here's a simple story for the emotional state of a pessimist. Not sure it's true but it's a story I have in mind.

In disaster movies, everything happens all at once. In reality climate disasters are more gradual, tsunamis and hurricanes and so on happen at different places at different times, but suppose for this story the disasters caused by climate change are all happening in a single period stretching on a few months. 100s of millions of people will be dead in a couple of months.

It's about a week before the disaster hits, and it's been shown very clearly (to most people's satisfaction) that it's about to hit.

Some people say "it is clearly time to do something, let us gather the troops and make lots of great plans". And that's true, in the world where there's work to be done. But it's also true that... the right time to solve this problem was 20+ years prior. Then there were real opportunities you could take that would have prevented the entire thing. Today there is no action that can prevent it, even though the threat is in the future. Fortunately it is not an existential threat, and there is triage to be done, and marginal effort can reliably save lives. But the time to solve it was decades ago, the opportunities lay unused, they were lost to time, and the catastrophe is here.

Returning to the reality we ourselves live in, insofar as someone is like "Huh, seems like these companies are producing what will soon be wildly lucrative products, and the search mechanism of gradient descent seems like it has the potential to produce a generally capable agent, and given the competitive nature of our economy and lack of political coordination it doesn't seem like we can stop lots of people from just pouring in money and having high-IQ people scale up their search mechanisms and potentially build an AGI — this is a big problem" I would say, hopefully without fear of contradiction, that the best time to avoid this catastrophe was decades ago, and that at this point the opportunities to avoid being in this situation have too been lost. 

(When reading the description above, I feel that Quirinus Quirrell would say the right thing to do is not get into that situation, and head it off way earlier in the timeline, because it sounds like an extremely doomed situation that required a miracle to avert catastrophe, and plans that successfully cause a miracle to happen are pretty fragile and typically fail.)

I'm not saying this story about being a week before disaster is fully analogous to the one we're in, but it's one I've been thinking about, and I think has been somewhat missing when people talk about the emotional state of pessimists.

In an attempt to make optimists take the problem more seriously, pessimists in this debate (and in particular EY) convinced themselves that all these paths are non-viable.

I feel like this is really not the correct causal story? My sense is that 'pessimists' evaluated those strategies hoping to find gold, and concluded they were just pyrite, and it was basically not motivated cognition, like this seems to imply.

Amusingly, MacKay often redraws the cartoon with the edits, so having 'biodiversity collapse' (the original purple wave) being surpassed by a fifth AGI wave seems like it reflects the situation more (tho, ofc, it requires more art effort ;) ).

A few thoughts:

1. "p(DOOM | Action)" seems too coarse a view from which to draw any sensible conclusions. The utility of pessimism (or realism, from some perspectives) is that it informs decisions about which action to take. It's not ok that we're simply doing something.
   Yes, the pessimistic view has a cost, but that needs to be weighed against the (potential) upside of finding more promising approaches - and realising that p(DOOM | we did things that felt helpful) > p(DOOM | we did things that may work).
2. Similarly, convincing a researcher that a [doomed-to-the-point-of-uselessness] approach is doomed is likely to be net positive, since they'll devote time to more worthwhile pursuits, likely increasing p(AWESOME...). Dangers being:
   1. Over-confidence that it's doomed.
   2. Throwing out wrong-but-useful ideas prematurely.
   3. A few researchers may be discouraged to the point of quitting.
3. In general, the best action to take will depend on p(Doom); if researchers have very different p(Doom) estimates, it should come as no surprise that they'll disagree on the best course of action at many levels.
   
   Specifically, the upside/downside of 'pessimism' will relate to our initial p(Doom). If p(Doom) starts out at >99.9%, then p(Doom | some vaguely plausible kind of action) is likely also >99%. High values of p(Doom) result from the problem's being fundamentally difficult - with answers that may be hidden in a tiny region of a high-dimensional space.
   If the problem is five times as hard as we may reasonably hope, then a shotgun may work.
   If the problem is five orders of magnitude harder, then we want a laser.
   1. Here I emphasize that we'd eventually need the laser. By all means use a shotgun to locate promising approaches, but if the problem is hard we can't just babble: we need to prune hard.
   2. It occurs to me that it may be useful to have more community-level processes/norms to signal babbling, so that mistargeted pruning doesn't impede idea generation.
4. Pivotal act talk does seem to be disastrous PR, but I think that's the only sense in which it's negative. By Critch's standards, the arguments in that post are weak. When thinking about this, it's important to be clear what we're saying on the facts, and what adjustments we're making for PR considerations. React too much to the negative PR issue (which I agree is real), and there's the danger of engaging in wishful thinking.
   1. Should we try to do tackle the extremely hard coordination problems? Possibly - but we shouldn't be banking on such approaches working. (counterfactual resource use deserves consideration)
   2. We should be careful of any PR-motivated adaptations to discourse, since they're likely to shape thinking in unhelpful ways. (even just talking less about X, rather than actively making misleading claims about X may well have a negative impact on epistemics)

P(DOOM from AGI | No Action) = 0. We will not die by AGI if we take no action to develop AGI, at least in less than astronomical timescales.

This is obviously not what will happen. Many groups will rush to develop AGI in the fastest and most powerful manner that they can possibly arrange, with safety largely treated as a public relations exercise and not a serious issue. My personal feeling is that any AGI developed in the next 30 years will very likely (p > 0.9) be sufficiently misaligned that humanity would be extinguished if it were scaled up to superintelligence.

My hopes for our species rest on one of three foundations: that I'm hopelessly miscalibrated about how hard alignment is, that AGI takes so much longer to develop that alignment is adequately solved, or that scaling to superintelligence is for some reason very much harder than developing AGI.

The second seems less likely each year, and the third seems an awfully flimsy hope. Both algorithm advances and hardware capabilities seem to offer substantial improvements well into the superhuman realm for all of the many narrow tasks that machines can now do better than humans, and I don't see any reason to expect that they cannot scale at least as well for more general ones once AGI is achieved.

I do have some other outlier possibilities for hope, but they seem more based in fiction than reality.

So that mainly leaves the first hope, that alignment is actually a lot simpler and easier than it appears to me. It is at least worth checking.

From all the stuff I've been reading here lately, it seems the only solution would be total worldwide oversight control of all activities.