LessWrong 2.0 Reader

Let's list some mysteries in this case! Here are 3 of mine.

1. Why did 2 killings happen within the span of one week?
2. How did so many people get radicalized i.e. going from being essentially nerds to people seemingly willing to commit pre-meditated murder?
3. What happened with Zajko and LaSota between Zajko writing about LaSota threatening to murder Zajko, and then Zajko's parent’s killings (which she is a suspect for)?

There are a lot of journalists and documentarians who are inquiring about this, wanting to write articles and make documentaries. 

What are people's heuristics for how to speak well with journalists and to choose whether and which journalists to talk with? Here are two that I've heard:

1. Small-town journalists tend to be much less politicized than those who write for a national outlet.
2. It's good to always be "off the record" when speaking, and then say you're happy to provide written quotes on-the-record via email afterward. This means you can (a) be deliberate in your wording, and (b) have proof if they misquote you!

Well I appreciate your comment but I think something's missing as far as conveying the emotions of the situation. I can imagine a death, a car crash for example, or imagine death on an even bigger scale like a nuclear weapon. I can imagine a disaster movie before it resolves on a happy ending. But I think those conceptions don't convey much, because I acknowledge that superintelligence can be destructive and can even envision what the end state of destruction would look like. Just envisioning that end state without explaining superintelligence that caused us to get to that end state doesn't do much for me though. 

The problem with this neat picture is reward-hacking. This process wouldn't optimize for better performance on fuzzy tasks, it would optimize for performance on fuzzy tasks that looks better to the underlying model. And much like RLHF doesn't scale to superintelligence, this doesn't scale to superhuman fuzzy-task performance.

It can improve the performance a bit. But once you ramp up the optimization pressure, "better performance" and "looks like better performance" would decouple from each other and the model would train itself into idiosyncratic uselessness. (Indeed: if it were this easy, doesn't this mean you should be able to self-modify into a master tactician or martial artist by running some simulated scenarios in your mind, improving without bound, and without any need to contact reality?)

... Or so my intuition goes. It's possible that this totally works for some dumb reason. But I don't think so. RL has a long-standing history of problems with reward-hacking, and LLMs' judgement is one of the most easily hackable things out there.

(Note that I'm not arguing that recursive self-improvement is impossible in general. But RLAIF, specifically, just doesn't look like the way.)

Small but repeated error: you mean “Ginkgo Bioworks”, right?

Thanks, that's a very helpful way of putting it!

Not having thought about it for very long, my intuition says "minimizing the description length of $A\left(x\right)$ definitely shouldn't impose constraints on the components themselves," i.e. "Alice has no use for the rank-1 attributions." But I can see why it would be nice to find a way for Alice to want that information, and you probably have deeper intuitions for this.

Thank you for sharing negative results!! 

Thanks for this.  Discussions of things like "one time shifts in power between humans via mechanisms like states becoming more powerful" and personal AI representatives is exactly the sort of thing I'd like to hear more about.  I'm happy to have finally found someone who has something substantial to say about this transition!

But over the last 2 years I asked a lot of people at the major labs about for any kind of details about a positive post-AGI future and almost no one had put anywhere close to as much thought into it as you have, and no one mentioned the things above.  Most people clearly hadn't put much thought into it at all.  If anyone at the labs had much more of plan than "we'll solve alignment while avoiding an arms race", I managed to fail to even hear about its existence despite many conversations, including with founders.

The closest thing to a plan was Sam Bowman's checklist:
https://sleepinyourhat.github.io/checklist/
which is exactly the sort of thing I was hoping for, except it's almost silent on issues of power, the state, and the role of post-AGI humans.

If you have any more related reading for the main "things might go OK" plan in your eyes, I'm all ears.

Unlike the Hobbes snippet, I didn’t feel like the Hume excerpt needed much translation to be accessible. I think I would decide on a case-by-case basis whether to read the translated version or the original rather than defaulting to one or the other.

Nod. I'm just answering your question of why I consider it optimistic.