LessWrong 2.0 Reader

Sure, it sounds like a good idea! Below I will write my thoughts on your overall summarized position.

———

"I primarily wish to argue that, given the general lack of accountability for developing machine learning systems in worlds where indeed the default outcome is doom, it should not be surprising to find out that there is a large corporation (or multiple) doing so."

I do think that I could maybe agree with this if it was 1 small corporation. In your previous comment you suggested that you are describing not the intentional contribution to the omnicide, but the bit of rationalization. I don't think I would agree that that many people working on AI are successfully engaged in that bit of rationalization or that it would be enough to keep them doing it. The big factor is that in case of their failure, they personally (and all of their loved ones) will suffer the consequences.

"It is also not surprising that glory-seeking companies have large departments focused on 'ethics' and 'safety' in order to look respectable to such people."

I don't disagree with this, because it seems plausible that one of the reasons for creating safety departments is ulterior. However, I believe that this reason is probably not the main one and that AI safety labs are making genuinely good research papers. To take an example of Anthropic, I've seen safety papers that got LessWrong community excited (at least judging by upvotes). Like this [LW · GW] one.

"I believe that there is no good plan and that these companies would exist regardless of whether a good plan existed or not... I believe that the people involved are getting rich risking all of our lives and there is (currently) no justice here"

For the reasons, that I mentioned in my first paragraph I would probably disagree with this. Relatedly, while I do think wealth in general can be somewhat motivating, I also think that AI developers are aware that all their wealth would mean nothing if AI kills everyone.

———

Overall, I am really happy with this discussion. Our disagreements came down to a few points and we agree on quite a bit of issues. I am similarly happy to conclude this big comment thread.

Good write up! People Cannot Handle "fill in the blank" on smartphones. Sex, food, drugs, social status, betting, binge watching, shopping etc. in abundance and a click away is something we cannot not handle. If some of biggest corporations in the world spends billions upon billions each year to grab our attention, they will win and "you" will on average loose, unless you pull the cord (or turn off the wifi...) or have extreme will power. 

I am definitely not the one to throw the first rock, but is it not pretty embarrassing that most of us who thought we were so smart and independent are mere serfs, both intellectually and physically, to a little piece of electronics that have completely and utterly hijacked our brains and bodies.

Related: https://www.lesswrong.com/posts/vcuBJgfSCvyPmqG7a/list-of-collective-intelligence-projects [LW · GW]

I had never thought about approaching this topic from the abstract, but I'm judging from the karma that this is actually what people want, rather than existing projects.

I'm surprised! I thought people were overall disinterested about this topic, but it seems more like the problem itself hadn't been stated to start with.

if you ask mathematicians whether ZFC + not Consistent(ZFC) is consistent, they will say "no, of course not!"

I suspect than many people's intuitive interpretation of "consistent" is ω-consistent, especially if they are not aware of the distinction.

I find it difficult to make distinct categories, but there seem to be two dimensions along which to classify relations:

1. How intense is the relation / how much we "click" emotionally and intellectually.
2. Whether the relation is expected to survive the change of current context.

(Even this is not a clear distinction, because "my relatives" is kinda contextual, but the context is there forever.)

Mapping to your system: close friends = high intensity context independent; friendly acquaintances = high intensity contextual; acquaintances = low intensity contextual.

One quadrant seems to be missing, but maybe that makes sense: if the relation is low intensity, why would people bother to keep it outside of the context where it originated.

It's a great idea. I ended up bolding the one line that states my conclusion to make it easier to spot.

Strong agree.

Hiding the main result beyond spoiler makes for a great reveal, but less useful for skimming.

That's crazy how close that is. $e=\sqrt[3]{32}0$ (to the nearest half a percent) will be a fun fact that I remember now!

Thanks for the Quenya tip. I tried Artano and it didn't work very quickly. Given that apparently it does in fact work, I can try that again. 

Creating an inhumanly good model of a human is related to formulating their preferences.

How does this relate to my idea? I'm not talking about figuring out human preferences.

Thus it's a step towards eliminating path-dependence of particular life stories

What is "path-dependence of particular life stories"?

I think things (minds, physical objects, social phenomena) should be characterized by computations that they could simulate/incarnate.

Are there other ways to characterize objects? Feels like a very general (or even fully general) framework. I believe my idea can be framed like this, too.