LessWrong 2.0 Reader

Sad to hear. Is this thread itself (starting with my parent comment which you replied to) an example of this, or are you referring instead to previous engagements/threads on LW?

Right, thus the large sales force. Standard B2B business model where the product is mediocre but there's a strong sales team convincing idiots in suits to pay ridiculous amounts of money for it.

And here I was hoping it would prompt someone to look things up or talk about them with the person who recommended the book.

I definitely agree that under the more common usage of safety that an AI doing what a human ordered in taking over the world or breaking laws for their owner would not be classified as safe, but in an AI safety context, alignment/safety does usually mean that these outcomes would be classified as safe.

My own view is that the technical problem is IMO shaping up to be a relatively easy problem, but I think that the political problems of advanced AI will probably prove a lot harder, especially in a future where humans control AIs for a long time.

Making money at all in biology requires being a therapeutics company, which requires you to do something exciting

Illumina has a market cap of 22,77 billion. There was a time when Theranos had a high market cap even if they ultimately didn't manage to develop the technology for it. 

It's possible to make a lot of money building tools, it's just that most of the capital is therapeutics-focused instead of tool-focused. However, theraputics-focus vs. tool focused is not the same thing as boring/interesting. Neither Illumina nor Theranos are boring. Alpha Fold was exciting but there's still a reason why it was developed at Google and not at a big pharma company.

If we look at the question of incubators, there's probably a company that sells the incubators and the software that runs them is closed-source so it's hard for someone besides the incubator company to provide software to control it. 

The first sales page I found for an incubator is https://www.thermofisher.com/order/catalog/product/51031528?SID=srch-srp-51031528 . If you want to create an incubator startup, building an incubator that can do all the things that the incubator from Thermo Fisher can do and additionally has WLan and an app, you have to do a lot of work to match the features of the existing incubator. Even if you could produce the product, I expect it will not easy to sell it and get people to trust you to have a better product than Thermo Fisher. 

Thermo Fisher likely does market analysis and would build build an app for their incubator if they would think that their customers want that but currently sees no demand. 

It might be inherent, in idea of having an app to control the incubator being boring, that it's hard to sell it incubators with it. 

Thanks! I should have been more clear that the trajectory toward level 5 (with all human virtue/trust being hackable for instrumental gains) itself is concerning, not just the eventual leap when it gets there.

This chain of logic is founded on an assumption that these technologies are possible, which I find highly dubious. If an (aligned) superintelligence is built, and we ask it for life extension, the most probable answer would be that biological immortality (and all stuff requiring nanorobots) is just plain impossible, and brain uploading wouldn't help because your copy is not you.

Somehow Docusign got the Swiss government to pay them a lot of money for providing e-signatures [LW · GW] instead of that service provided order of magnitudes cheaper by a startup with two full time developers. There are no companies who use the existence of AWS to do disruptive innovation to eat Docusigns profits away. 

I think I agree with this idea in principle, but I also feel like it misses some things in practice (or something). Some considerations:

• I think my bar for "how much I trust a lab such that I'm OK with them not making transparency commitments" is fairly high. I don't think any existing lab meets that bar.
• I feel like a lot of forms of helpful transparency are not that costly. The main 'cost' feels to me like "maybe the government will end up regulating the sector if/when it understands how dangerous industry people expect AI systems to be and how many safety/security concerns they have". But I think things like "report dangerous stuff to the govt", "have a whistleblower mechanism", and even "make it clear that you're willing to have govt people come and ask about safety/security concerns" don't seem very costly from a time/effort perspective.
• If a Responsible Company implemented transparency stuff unilaterally, it would make it easier for the government to have proof-of-concept and implement the same requirements for other companies. In a lot of cases, showing that a concept works for company X (and that company X actually thinks it's a good thing) can reduce a lot of friction in getting things applied to companies Y and Z.

I do agree that some of this depends on the type of transparency commitment and there might be specific types of transparency commitments that don't make sense to pursue unilaterally. Off the top of my head, I can't think of any transparency requirements that I wouldn't want to see implemented unilaterally, and I can think of several that I would want to see (e.g., dangerous capability reports, capability forecasts, whistleblower mechanisms, sharing if-then plans with govt, sharing shutdown plans with govt, setting up interview program with govt, engaging publicly with threat models, having clear OpenAI-style tables that spell out which dangerous capabilities you're tracking/expecting).

This would seem like a great benchmark/dataset/eval to apply automated research to [LW(p) · GW(p)]. Would you have thoughts/recommendations on that?