LessWrong 2.0 Reader

Good to know. I also quoted your more detailed remark on AI Standards Lab at the top of this post.

I have made so many connections that have been instrumental to my research. 

I didn't know this yet, and glad to hear!  Thank you for the kind words, Nell.

Related question: What is the least impressive game current LLMs struggle with?

I’ve heard they’re pretty bad at Tic Tac Toe.

Maybe we can draw a line between the score an AI gets without using human written problem/solution pairs in any way, and the score an AI gets after using them in some way (RL on example questions, training on example solutions, etc.).

In the former case, we're interested in how well the AI can do a task as difficult as the test, all on its own. In the latter case, we're interested in how well the AI can do a task as difficult as the test, if working with humans training it for the task.

I really want to make it clear I'm not trying to badmouth o3, I think it is a very impressive model. I should've written my post better.

Progress in ML looks a lot like, we had a different setup with different data and a tweaked algorithm and did better on this task. If you want to put an asterisk on o3 because it trained in some specific way that's different from previous contenders, then basically every ML advance is going to have a similar asterisk. Seems like a lot of asterisking.

I'm not sure what the details would look like, but I'm pretty sure ASI would have enough new technologies to figure something out within 10,000 years.

I feel like this is the main load-bearing claim underlying the post, but it's barely argued for.

In some sense the sun is already "eating itself" by doing a fusion reaction, which will last for billions more years. So you're claiming that AI could eat the sun (at least) six orders of magnitude faster, which is not obvious to me.

I don't think my priors on that are very different from yours but the thing that would have made this post valuable for me is some object-level reason to upgrade my confidence in that.

Trying this again I think the question sets need a little work.

Concrete existence, they point out, is less resource efficient than dreams of the machine. Hard to tell how much value is tied up in physical form and not computation, if humans would agree on this either way on reflection.

Perhaps you'd be interested in adding a page on AI Safety & Entrepreneurship?

Curated! I was hesitant about this one for a while because Ray is staff, but I do think it ultimately is doing a quite impressive thing where it is at least a bit successfully communicating a whole worldview and perspective on the art of rationality, and that kind of content I think makes up a large fraction of the most impactful writing on LessWrong. 

It does have the problem of being long, and not necessarily keeping its pace all-throughout. I do recommend that readers feel free to skip sections and start skimming as they get bored. Maybe that's wrong, but it helped me.

I like the post a lot for being honest in a way that few other writing is. I think it in some contributes to its somewhat meandering nature, but I do feel like while reading the post my mind learns a bit more to conform itself to the shape of Ray's thoughts, which at the very least is helpful for modeling Ray, but also helpful in as much as Ray has learned at least some things that you the reader have not yet learned, or in as much as you are facing problems for which Ray's mind is better suited than your own, which I do think is probably true.