LessWrong 2.0 Reader

Are you suggesting that I should base my morality on whether I'll be rewarded for adhering to it? That just sounds like selfishness disguised as impersonal ethics.

To be clear, I do have some selfish preferences. I care about my own life and happiness, and the happiness of my friends and family. But I also have some altruistic preferences, and my commentary on AI tends to reflect that.

Have you looked at samples of CoT of o1, o3, deepseek, etc. solving hard math problems?

Certainly (experimenting with r1's CoTs right now, in fact). I agree that they're not doing the brute-force stuff I mentioned; that was just me outlining a scenario in which a system "technically" clears the bar you'd outlined, yet I end up unmoved (I don't want to end up goalpost-moving).

Though neither are they being "strategic" in the way I expect they'd need to be in order to productively use a billion-token CoT.

Anyhow, this is nice, because I do expect that probably something like this milestone will be reached before AGI

Yeah, I'm also glad to finally have something concrete-ish to watch out for. Thanks for prompting me!

The question is "are the safety-conscious people effectual at all, and what are their opportunity costs?".

i.e. are the cheap things they can do that don't step on anyone's toes that helpful-on-the-margin, better than what they'd be able to do at another company? (I don't know the answer, depends on the people).

It does seem to imply that, doesn't it? I respect the people leaving, and I think it does send a valuable message. And it seems very valuable to have safety-conscious people on the inside.

"I don't think you'll need to worry about this stuff until you get really far out of distribution." I may sound like I'm just commenting for the sake of commenting but I think that's something you want to be crystal clear on. I'm pessimistic in general and this situation is probably unlikely but I guess one of my worst fears would be creating uberpsychosis. Sounding like every LWer, my relatively out of distribution capabilities made my psychotic delusions hyper-analytic/1000x more terrifying & elaborate than they would have been with worse working memory/analytic abilities (once I started ECT I didn't have the horsepower to hyperanalyze existence as much). I guess the best way to describe it was that I could feel the terror of just how bad -inf would truly be as opposed to having an abstract/detached view that -inf = bad. And I wouldn't want anyone else to go through something like that, let alone something much scarier/worse. 

if you get enough meditative insight you'll transcend the concept of a self

What is the notion of self that you transcend, what does it mean to transcend it, and how does meditation cause this to happen? 

Have you looked at samples of CoT of o1, o3, deepseek, etc. solving hard math problems? I feel like a few examples have been shown & they seem to involve qualitative thinking, not just brute-force-proof-search (though of course they show lots of failed attempts and backtracking -- just like a human thought-chain would).

Anyhow, this is nice, because I do expect that probably something like this milestone will be reached before AGI (though I'm not sure)

Does this imply that fewer safety people should quit leading labs to protest poor safety policies?

See also the classic LW post, The Best Textbooks on Every Subject [LW · GW].

I have significant [LW · GW] meditative insight [LW · GW].

I feel that the quotes you use to describe Camp #1 and Camp #2 are both word salad. The Camp #1 quote is like this post [? · GW] Eliezer Yudkowsky wrote 17 years ago, except difficult to understand^[1]. Camp #2 asserts the existence of something fundamental, and then follows it with "There's no agreement on what this thing is". I feel that these postulations are not well-defined enough to deserve refutation.

How have your personal experiences influenced your understanding of western philosophy of mind. Not only on the topic of qualia; that was just the example that motivated me to post the question. For example, did anyone move from Harth's camp #1 to camp #2 or vice versa after meditation experiences, or did any of your other philosophical positions shift?

I feel that the mainstream western philosophy of mind is a train wreck, and that this is obvious to anybody who is half-decent at writing clearly. This includes both Eliezer Yudkowsky and Paul Graham, neither of whom (to my knowledge) have significant meditative insight. This question is like asking "Did you stop eating garbage out of the local Safeway dumpster after you learned to cook?" I was never eating it to begin with.

To be fair, Buddhist metaphysics isn't any better. It's not uncommon for meditators with deep insight to also believe in levitation and reincarnation.

I think what you're really trying to ask is "Do you have any personal observations via insight which contradict a major philosophical school?" Yes, if you get enough meditative insight you'll transcend the concept of a self. Anything system of philosophy that begins with "I think, therefore I am" is broken at almost the axiomatic level.