LessWrong 2.0 Reader

@ryan_greenblatt [LW · GW] can you say more about what you mean by this one?

1. We can't find countermeasures such that our control evaluations indicate any real safety.

I appreciate this distinction between the different types of outcomes-- nice. 

I'm pretty interested in peoples' forecasts for each. So revised questions would be:

1a) Using your best judgment, at what point do you expect sabotage evaluations will no longer be able to reliably/confidently help us understand whether or not a model is scheming? At what capability threshold do you expect we would no longer be able to say "yup, the model is not scheming, and we can be confident that our sabotage evals are accurate."? (Feel free to use heuristics like the OpenAI PF capability thresholds or ASL levels).

1b) Using your best judgment, how likely is it that sabotage evaluations are undermined such that humans think the model can be deployed safely, but actually the humans were misled? At what capability threshold would you worry about this form of undermining?

@David Duvenaud [LW · GW] would be curious about your takes, even if they're just speculative guesses/predictions. 

(And @ryan_greenblatt [LW · GW] would be curious for your takes but replace "sabotage evaluations" with "control evaluations." In the event that you've written this up elsewhere, feel free to link it– helpful for me if you quote the particular section.)

Sad to hear. Is this thread itself (starting with my parent comment which you replied to) an example of this, or are you referring instead to previous engagements/threads on LW?

Right, thus the large sales force. Standard B2B business model where the product is mediocre but there's a strong sales team convincing idiots in suits to pay ridiculous amounts of money for it.

And here I was hoping it would prompt someone to look things up or talk about them with the person who recommended the book.

I definitely agree that under the more common usage of safety that an AI doing what a human ordered in taking over the world or breaking laws for their owner would not be classified as safe, but in an AI safety context, alignment/safety does usually mean that these outcomes would be classified as safe.

My own view is that the technical problem is IMO shaping up to be a relatively easy problem, but I think that the political problems of advanced AI will probably prove a lot harder, especially in a future where humans control AIs for a long time.

Making money at all in biology requires being a therapeutics company, which requires you to do something exciting

Illumina has a market cap of 22,77 billion. There was a time when Theranos had a high market cap even if they ultimately didn't manage to develop the technology for it. 

It's possible to make a lot of money building tools, it's just that most of the capital is therapeutics-focused instead of tool-focused. However, theraputics-focus vs. tool focused is not the same thing as boring/interesting. Neither Illumina nor Theranos are boring. Alpha Fold was exciting but there's still a reason why it was developed at Google and not at a big pharma company.

If we look at the question of incubators, there's probably a company that sells the incubators and the software that runs them is closed-source so it's hard for someone besides the incubator company to provide software to control it. 

The first sales page I found for an incubator is https://www.thermofisher.com/order/catalog/product/51031528?SID=srch-srp-51031528 . If you want to create an incubator startup, building an incubator that can do all the things that the incubator from Thermo Fisher can do and additionally has WLan and an app, you have to do a lot of work to match the features of the existing incubator. Even if you could produce the product, I expect it will not easy to sell it and get people to trust you to have a better product than Thermo Fisher. 

Thermo Fisher likely does market analysis and would build build an app for their incubator if they would think that their customers want that but currently sees no demand. 

It might be inherent, in idea of having an app to control the incubator being boring, that it's hard to sell it incubators with it. 

Thanks! I should have been more clear that the trajectory toward level 5 (with all human virtue/trust being hackable for instrumental gains) itself is concerning, not just the eventual leap when it gets there.

This chain of logic is founded on an assumption that these technologies are possible, which I find highly dubious. If an (aligned) superintelligence is built, and we ask it for life extension, the most probable answer would be that biological immortality (and all stuff requiring nanorobots) is just plain impossible, and brain uploading wouldn't help because your copy is not you.

Somehow Docusign got the Swiss government to pay them a lot of money for providing e-signatures [LW · GW] instead of that service provided order of magnitudes cheaper by a startup with two full time developers. There are no companies who use the existence of AWS to do disruptive innovation to eat Docusigns profits away.