LessWrong 2.0 Reader

Maybe it could be FLCI to avoid collision with the existing FLI.

Thanks as well, 

I will just say that I am not saying those things for social purposes, I am just stating what I think is true. And I am not baseless as there are studies that show how kantianism and superrationality can resolve cooperative issues and be optimal for agents. You seem to purely disregard these elements, as if they don't exist (it's how it feels from my perspective)

There are differences in human evolutions that show behavioral changes, we have been pretty cooperative, more than other animals, many studies show that human cooperate even when it is not in their best selfish interest. 

However, we (also) have been constructing our civilization on destruction. Nature is based on selection, which is a massacre, so it is 'pretty' coherent for us to inherit those traits. 

Despite that, we have seen many positive growth in ethics that increasingly fit with kantianism. 

Evolution takes time and comes from deep-dark places, to me a core challenge is to transition towards super-cooperation while being a system made of irrational agents, during polycrises. 


There is also a gap between what people want (basically everybody agrees that there are urgent issues to handle as a society, but almost all declare that "others won't change"; I know this because I've been conversing with people from all age/background for half my life on subjects related to crises). What people happen to do under pressure due to the context and constraints isn't what they'd want if things were different, if they have had certain crucial informations before etc. 

When given the tools, such as the moral graph procedure that has been tested recently, things change to the better in a clear and direct way. People initially diverging start to see new aspects on which they converge. There are other studies related to crowd wisdom showing that certain ingredients need to be put together for wisdom to happen (Surowiecki's recipe: Independence, Diversity and Aggregation). We are in the process of building better systems, our institutions are yet catastrophic on many levels. 



In the eyes of many, I am still very pessimistic, so the apparent wishful thinking is quite relative (I think it's an important point)

I also think that an irrational artificial intelligence might still have high causal impact, and that it isn't easy to be rational even when we 'want to' at some level, or that we see what should be the rational road empirically, yet don't follow it. Irreducibility is inherent to reality

Anyway despite my very best I might be irrational right now, 
We all are, but I might be more than you who knows? 

Can you give examples of what you're looking for? Can I email you entries and expect a response?

Your answer might also be "I, Oliver, will play this role". My gut take would be excited for you to be like one of three people in this role (with strong co-leads, who are maybe complementary in the sense that they're strong at some styles of thinking you don't know exactly how to replicate), and kind of weakly pessimistic about you doing it alone. (It certainly might be that that pessimism is misplaced.)

For what it’s worth, my guess is that your pessimism is misplaced. Oliver certainly isn’t as famous as Bostrom, so I doubt he’d be a similar “beacon.” But I’m not sure a beacon is needed—historically, plenty of successful research institutions (e.g. Bells Labs, IAS, the Royal Society in most eras) weren’t led by their star researchers, and the track record of those that were strikes me as pretty mixed.

Oliver spends most of his time building infrastructure for researchers, and I think he’s become quite good at it. For example, you are reading this comment on (what strikes me as) rather obviously the best-designed forum on the internet; I think the review books LessWrong made are probably the second-best designed books I’ve seen, after those from Stripe Press; and the Lighthaven campus is an exceedingly nice place to work.

Personally, I think Oliver would probably be my literal top choice to head an institution like this.

Thanks for the conversation and exploration!  I have to admit that this doesn't match my observations and understanding of power and negotiation in the human agents I've been able to study, and I can't see why one would expect non-humans, even (perhaps especially) rational ones, to commit to alliances in this manner.

I can't tell if you're describing what you hope will happen, or what you think automatically happens, or what you want readers to strive for, but I'm not convinced.  This will likely be my last comment for awhile - feel free to rebut or respond, I'll read it and consider it, but likely not post.

If we live in naive MWI, an IBP agent would not care for good reasons, because naive MWI is a “library of babel” where essentially every conceivable thing happens no matter what you do.

Isn't the frequency of amplitude-patterns changes depending on what you do? So an agent can care about that instead of point-states.

I ask partly because I personally would be more excited of a version of this that wasn't ignoring AGI timelines, but I think a version of this that's not ignoring AGI timelines would probably be quite different from the intellectual spirit/tradition of FHI.

This frame feels a bit off to me. Partly because I don’t think FHI was ignoring timelines, and I think their work has proved quite useful already—mostly by substantially improving our conceptual vocabulary for reasoning about things like existential risk.

But also because it feels important to me to stress that the portfolio of alignment research with maximal expected value need not necessarily perform well in the most likely particular world. One might imagine, for example—and indeed this is my own bet—that the most valuable actions we can take will only actually save us in the subset of worlds in which we have enough time to develop a proper science of alignment.

Yeah, that might be a good idea in case any rich employers stumble on this :-)

In terms of goals, I like making something, having many people use it, and getting paid for it. I'm not as motivated by meaning, probably different from most EAs in that sense.

In terms of skillset, I'd say I'm a frontend-focused generalist. The most fun programming experience in my life was when I built an online map just by myself - the rendering of map data to png tiles, the serving backend, the javascript for dragging and zooming, there weren't many libraries back then - and then it got released and got hundreds of thousands of users. The second most fun was when I made that game - the idea, iterating on the mechanics, graphic design, audio programming, writing, packaging for web and mobile, the whole thing - and it got quite popular too. So that's the prototypical good job for me.

I don't really understand your approach yet. Let's call your decision theory CLDT. You say counterfactuals in CLDT should correspond to consistent universes. For example, the counterfactual "what if a CLDT agent two-boxed in Newcomb's problem" should correspond to a consistent universe where a CLDT agent two-boxes on Newcomb's problem. Can you describe that universe in more detail?

I think you are slightly muddling your phrases.

You are richer if you can afford more goods and better goods. But not all goods will necessarily change price in the same direction. Its entirely possible that you can become richer, but that food prices grow faster than your new income. (For example, imagine that your income doubles, that food prices also double, but prices of other things drop so that inflation remains zero. You can afford more non-food stuff, and the same amount of food, so you are richer overall. This could happen even if food prices had gone up faster than your income.)

I think a (slightly cartoony) real life example is servants. Rich people today are richer than rich people in Victorian times, but fewer rich people today (in developed countries) can afford to have servants. This is because the price of hiring servants has gone up faster than the incomes of these rich people. So it is possible for people to get richer overall, while at the same time some specific goods or services become less accessible.

Maybe a more obvious example is rent (or housing in general). A modern computer programmer in Silicon valley could well be paying a larger percentage of their income on housing than a medieval peasant. But, they can afford more of other things than that peasant could.