LessWrong 2.0 Reader

I agree with your point! That's why I started with the word "theoretically".

Rob Miles also makes the point that if you expect people to accurately model the incoming doom, you should have a low p(doom). At the very least, worlds in which humanity is switched-on enough (and the AI takeover is slow enough) for both markets to crash and the world to have enough social order for your bet to come through are much more likely to survive. If enough people are selling assets to buy cocaine for the market to crash, either the AI takeover is remarkably slow indeed (comparable to a normal human-human war) or public opinion is so doomy pre-takeover that there would be enough political will to "assertively" shut down the datacenters.

At least this has helped clarify that you think of S(A) to (often) precede A by a lot, which wasn't clear to me.

Not really; instead, I think throwing the ball is a time-extended course of action, as most actions are. If I “decide” to say a sentence or sing a song, I don’t separately “decide” to say the next syllable, then “decide” to say the next syllable after that, etc.

What do you make of the Libet experiments?

He did a bunch of experiments, I’m not sure which ones you’re referring to. (The “conscious intentions” one?) The ones I’ve read about seem mildly interesting. I don’t think they contradict anything I wrote or believe. If you do think that, feel free to explain. :)

Problems with that.

Doom doesn't imply that everyone believes in doom before it happens.

Do you think that the evidence for doom will be more obvious than the evidence for atheism, while the world is not yet destroyed?

It's quite possible for doom to happen, and most people to have no clue beyond one article with a picture of red glowing eyed robots. 

If everyone does believe in doom, there might be a bit of spending on consumption. But there will also be lots of riots, lots of lynching and burning down data centers and stuff like that. 

In this bizarre hypothetical where everyone believes doom is coming soon and starts enjoying their money while they can, then society is starting to fall apart, and the price of luxuries is through the roof. Your opportunities to enjoy the money will be limited. 

It should! Fixed, thank you :)

Well, it's not exactly going to be a surprise to the LLMs ... it may feel more like knowing for certain we are a servant species uplifted by cats.

We’re definitely talking past each other somehow.

I guess this will only stop when we have made our thoughts clear enough for an implementation that allows us to inspect the system for S(A) and A. Which is OK.

At least this has helped clarify that you think of S(A) to (often) precede A by a lot, which wasn't clear to me. I think this complicates the analysis because of where to draw the line. Would it count if I imagine throwing the ball one day (S(A)) but executing it during the game the next day as I intend? 

What do you make of the Libet experiments?

If system prompts aren't enough but fine-tuning is, this should be doable with different adapters that can be loaded at inference time; not needing to distill into separate models.

We’re definitely talking past each other somehow. For example, your statement “The S(A) is precisely timed to coincide with the release” is (to me) obviously false. In the case of “deciding to throw a ball”, A would be the time-extended action of throwing the ball, and S(A) would be me “making a decision of my free will” to throw the ball, which happens way before the release, indeed it happens before I even start moving my arm. Releasing the ball isn’t a separate “decision” but rather part of the already-decided course-of-action.

(Again, I’m definitely not arguing that every action is this kind of stereotypical [S(A); A] “intentional free will decision”, or even that most actions are. Non-examples include every action you take in a flow state, and indeed you could say that every day is full of little “micro-flow-states” that last for even just a few seconds when you’re doing something rather than self-reflecting.)

…Then after the fact, I might recall the fact that I released the ball at such-and-such moment. But that thought is not actually about an “action” for reasons discussed in §2.6.1.

"I really don't know how tractable it would be to pressure compagnies" seems weirdly familiar.  We already used the same argument for AGI safety, and we know that governance work is much more tractable than expected.