LessWrong 2.0 Reader

Yes, I totally agree. But I kind of got distracted with this post and wanted to get back to work as quickly as possible. But instead of making it a perpetual draft I pushed a bit further and got some MVP thing. I agree it's not that good, and adding concrete examples, and really making this more like a tutorial would be the first step I would take.

So I am wondering if doing something like this is still useful. Would it be worse if I had made it a perpetual draft or better?

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2025. The top fifty or so posts are featured prominently on the site throughout the year. Will this post make the top fifty?

I don’t know about making god software, but human software is a lot of trial and error. I have been writing code for close to 40 years. The best I can do is write automated tests to anticipate the kinds of errors I might get. My imagination just isn’t as strong as reality.

There is provably no way to fully predict how a software system of sufficient complexity. With careful organization it becomes easier to reason about and predict, but unless you are writing provable software (it’s a very slow and complex process, I hear), that’s the best you get.

I feel you on being distracted by software bugs. I’m one of those guys that reports them, or even code change suggestions (GitHub Pull Requests).

But... Why would p(doom) move like Brownian motion until stopping at 0 or 1?

I don't disagree with your conclusions, there's a lot of evidence coming in, and if you're spending full time or even part time thinking about alignment, a lot of important updates on the inference. But assuming a random walk seems wrong.

Is there a reason that a complex, structured unfolding of reality would look like a random walk?

For what it's worth, being sedated for a wisdom tooth extraction preceded my stream entry (Buddhist term for an early stage of awakening) by about 6 months. Before that I had no real experience with being in severely altered states (other than accidentally Robotripping which I was unintentionally doing because I didn't realize how sensitive I was for DXM and had been experiencing it since I was a toddler taking cold medicine and thought it was just part of being sick). The experience of seeing myself continue to operate when "I" wasn't there was eye-opening.

"If you are assuming Software works well you are dead" because:

• If you assume this you will be shocked by how terrible software is every moment you use a computer, and your brain will constantly try to fix it wasting your time.
• You should not assume that humanity has it in it to make the god software without your intervention.
• When making god software: Assume the worst.

Also, radars are good in pinpointing exact coordinates in space and time. Optical recognition may have delays or difficultly in measuring distance. Even 1 second delay makes their information useless for supersonic aircrafts.

100 times more parameter efficient (102 vs 104 parameters) [this must be a typo, this would only be 1.01 times more parameter efficient].

clearly, they mean 10^2 vs 10^4. Same with the "10−7 vs 10−5 MSE". Must be some copy-paste/formatting issue.

Possibly a nitpick, but:

The development and deployment of AGI, or similarly advanced systems, could constitute a transformation rivaling those of the agricultural and industrial revolutions.

seems like a very strong understatement. Maybe replace "rivaling" with e.g. "(vastly) exceeding"?

I mean, sure, but I've been updating in that direction a weirdly large amount.