LessWrong 2.0 Reader

Not a biologist but my impression is that a lot of progress in biology came from refining and validating existing techniques. Also building up a large library of biological specimens & phenomena, i.e. taxonomy. The esthetic and practice of MechInterp seems in accord with that.

Yes, you make a good point. This post was written when our (my) understanding was less-developed. You might be interested in taking a look at Kolmogorov Algorithmic Sufficient Statistics for a more sophisticated framework that make the noise-structure decomposition in a principled way.

It seems worth mentioning that punishments for financial crime often include measures like "person gets banned from their industry" or them getting banned from participating in all kinds of financial schemes... But in theory, I like the idea of adding things to the sentencing that make re-offending less likely. This way, you can maybe justify giving people second chances. 

Good point. I can imagine things like "permanent parole" (note that probation and parole are meaningfully different) or being under house arrest or having constraints on your professional responsibilities or finances or something, being far better than literal incarceration.

I agree there are people who do small amounts of damage to society, are caught, and do not reoffend. Then there are other people whose criminal activities will be most of their effect on society, will reliably reoffend, and for whom the incapacitation strongly works out positive in consequentialist terms. My aim would be to have some way of distinguishing between them.

The amount of evidence we have about Bankman-Fried's character is quite different than that of most con men, including from childhood and from his personal diary, so I hope we can have more confidence based on that. But a different solution is to not do any psychologizing, and just judge based on reoffending. See this section from the ACX post:

In 2001, the Dutch government passed a law allowing longer sentences for criminals with at least ten previous offense who were not good targets for rehabilitation (eg rejected or had already failed drug treatment). The law allowed judges to increase the typical sentence for petty theft (2 months) to a longer sentence (2 years). A quasi-experimental study found that property crime, though not violent crime, decreased by 25%. It’s not surprising that violent crime didn’t go down since the law was almost entirely deployed against thieves.

Vollaard found that the population affected was extremely criminal; they had an average of 31 past offenses, and on surveys they admitted to committing an average of 256 crimes per year (mostly shoplifting). Before the law was passed, they spent an average of four months per year in jail (probably 2 x 2 month sentences); afterwards, they spent two years in jail per crime.

I should add that Scott has lots of concerns about doing this in the US, and argues that properly doing this in the US would massively increase the incarcerated population. I didn't quite follow his concerns, but I was not convinced that something like this would be a bad idea on consequentialist grounds, even if the incarcerated population were to massively increase. (Note that I would support improving the quality of prisons to being broadly as nice as outside of prisons.)

I will try to write down my thoughts on these problems below:

1) The Coordination Problem

For any organization developing AI, failing to align it is just as dangerous—if not more so—than losing the AI race altogether. If an organization has already secured the resources needed to win the capabilities race and has a functioning alignment solution (two of the most challenging hurdles), I'd be confident that it can successfully implement that solution (which, in comparison, seems like the easiest part). The risks of failing to implement alignment solutions are essentially the same as the risks of not having an alignment solution in the first place:

If you don't have a working alignment solution, you die.

If you fail to implement to implement a working alignment solution, you die.

Companies spending considerable resources on creating a working solution to the alignment problem will have all the same reasons for actually implementing it.

2) The Power Distribution Problem

I wouldn't necessarily frame this as a problem. Consider a world where multiple entities control AI—this scenario appears quite a bit more problematic. As it stands, the US is seemingly at the forefront of the AI race. Do we really want China and Russia to develop their own AIs? Even more troubling is the idea of multiple individuals owning superhuman AI. Just one person bent on global vengeance could lead to catastrophic outcomes. I'd be much more inclined to trust the AI race's winner to act in humanity's best interest than to rely on the goodness of every individual AI owner (including the winner of the AI race).

If the winner of the AI race will not act in humanity's best interests, then we won't have the means to make him share AI with others.

If the winner of the AI race will act in humanity's best interests, then we won't want him to share AI with other agents who might not act in humanity's best interests.

3) The Economic Transition Problem

If AI is aligned with human values, there is no need for humans to retain economic control. AI would simply leverage our economic resources for the benefit of humanity.

Yes, they would not be made from mirror components!

If the clone behaves indistinguishably from the human it is based on, then there is simply nothing more to say. It doesn't matter what is going on inside.

Right, I agree on that. The problem is, "behaves indistinguishably" for how long? You can't guarantee whether it will stop acting that way in the future, which is what is predicted by deceptive alignment.

My greatest ambition is to create a fully trainable art of rationality that’s so good it gets taught to every high schooler in the country and bankrupts multiple industries that prey on irrational behavior in the process. Although it may seem impossible, the success of anti-smoking efforts against an extremely addictive product with a massive advertising industry suggests that it's achievable, and the fact that the Internet exists now and didn't exist then suggests it's even easier than that was.

Suppose Sam Bankman-Fried is imprisoned for 25 years. After that time, he will be a decent, law-abiding member of society, who is safe to release from prison.

I voted 75% because taken literally I think in 25 years AI will be so advanced that he won't have much of an ability to impact the world at all 🤓

(Otherwise 40%)

Lol, I am sorry about the fundraising email. It was really quite embarrassing.

(Context, a recent fundraising email I sent out to a bunch of old LessWrong accounts had unsubscribe links that pointed to localhost:3000 instead of lesswrong.com, which of course is the most important link not to break)