LessWrong 2.0 Reader

Formal processes are mostly beneficial 

It feels like you wrote this line first, and nothing you wrote above it was going to change that conclusion. You recite a litany of horrors, grand and mundane, and then ignore them all.

and they’re not going anywhere.

This may be true, but that's no reason to embrace cope the equivalent of "death is what gives life meaning."

what would your preferred state of the timelines discourse be?

My main recommendation would be, "Don't pin down probability distributions that are (significantly) more precise than seems justified." I can't give an exact set of guidelines for what constitutes "more precise than seems justified" (such is life as a bounded agent!). But to a first approximation:

• Suppose I'm doing some modeling, and I find myself thinking, "Hm, what feels like the right median for this? 40? But ehh maybe 50, idk…"
• And suppose I can't point to any particular reason for favoring 40 over 50, or vice versa. (Or, I can point to some reasons for one number but also some reasons for the other, and it’s not clear which are stronger — when I try weighing up these reasons against each other, I find some reasons for one higher-order weighing and some reasons for another, etc. etc.)
  • This isn’t a problem for every pair of numbers that occurs to us when estimating stuff. If I have to pick between, say, 2030 or 2060 for my AGI timelines median, it seems like I have reason to trust my (imprecise!) intuition^[1] that AI progress is going fast enough that 2060 is unreasonable.
• Then: I wouldn't pick just one of 40 or 50 for the median, or just one number in between. I'd include them all.

I totally agree that we can't pin down the parameters to high precision

I'm not sure I understand your position, then. Do you endorse imprecise probabilities in principle, but report precise distributions for some illustrative purpose? (If so, I'd worry that's misleading.) My guess is that we're not yet on the same page about what "pin down the parameters to high precision" means.

I think this sort of work is valuable because it introduces new, comprehensive-ish frameworks for thinking about timelines/takeoff

Agreed! I appreciate your detailed transparency in communicating the structure of the model, even if I disagree about the formal epistemology.

communicating the reasoning behind our beliefs in a more transparent way than a non-quantitative approach would

If our beliefs about this domain ought to be significantly imprecise, not just uncertain, then I'd think the more transparent way to communicate your reasoning would be to report an imprecise (yet still quantitative) forecast.

1. ^{^}

   I don't want to overstate this, tbc. I think this intuition is only trustworthy to the extent that I think it's a compression of (i) lots of cached understanding I've gathered from engaging with timelines research, and (ii) conservative-seeming projections of AI progress that pass enough of a sniff test. If I came into this domain with no prior background, just having a vibe of "2060 is way too far off" wouldn't be a sufficient justification, I think.

For example, in ancient Greece it would have been taboo to say that women should have the same political rights as men.

Would it have been taboo?  Or would people have just laughed at you?  (Paul Graham said, e.g.: "[O]bviously false statements might be treated as jokes, or at worst as evidence of insanity, but they are not likely to make anyone mad. The statements that make people mad are the ones they worry might be believed."  Also relevant: "I suspect the biggest source of moral taboos will turn out to be power struggles in which one side only barely has the upper hand. That's where you'll find a group powerful enough to enforce taboos, but weak enough to need them.")

Investigating taboos is the harder problem, so if you solve that, then that's probably sufficient.

Absolutely agree with everything you've said. The problem of balancing accountability and blamelessness is hard. All I can say is, let's look at how it plays out in the real word. Here, I think, few general trends can be observed as to when less rigid process and less accountability is used:

1. In highly complex areas (professors, researchers)
2. When creativity is at a premium (dtto, art?)
3. When unpredictability is high (emergency medicine, SRE, rescue, flight control, military?)
4. When the incentives at the high and low level are aligned (e.g. procurement requires more rigid process, because incentives to prefer friends/family are just too high)

Few examples:

When working at an assembly line, the environment is deliberately crafted to be predictable, creativity hurts rather than helps. The process is rigid, there's no leeway.

At Google, there are both SREs and software engineers. The former are working in a highly unpredictable environment, the latter are not. From my observation, the former have both much less rigid processes and are blamed much less when things go awry.

Military is an interesting example which I would one day like to look more deeply into... As far as I understand the modern western system of units having their own agency instead of blindly following orders can be tracked back to Prussia trying to cope with the defeat by Napoleon. The idea is that a unit gets an goal without the instructions of how to accomplish it. The unit can then act creatively. But while that's the theory, it's hard to implement even to this day. It does not work at all for low-trust armies (Russian army), but even where trust is higher, there tend to be hick-ups. (I've also heard that the OKR system in business management may be descended from this framework, but again, more investigation would be needed.)

I believe that the author will be notified and may see it.

Maybe OpenAI did something to prevent its AIs from being pro-Hamas, in order to keep the Trump administration at bay, but it was too crude a patch and now it's being triggered at inappropriate times. 

I also realized a lot of my personal philosophy was independently discovered on lesswrong when I was about 16. 

As for doubting your ideas - there exists a healthy level, but it perhaps actually increases as you age. A little less doubt is useful for exploration. 

Randomness does seem important to many algorithms. ET Jaynes argued that there is ~always a more clever and harder to find deterministic approach that outperforms any randomized approach. There do seem to be counter examples in some cases though, such as mixed strategies in competitive games. However, I guess the most well-studied version of this question is BPP versus P which is still open.  

DM me when you’re finished, I want to see what you come up with.

(This aligns with what I intended. I feel like my comment is making a fine point, even despite having missed the specific section.)

To me it seems that the burden of proof lies on the side that asserts that human minds are able to access all heights of creation. Why should that be? Are we past some specific threshold?