LessWrong 2.0 Reader

Interesting! 

I definetely see your point in how the incentives here are skewed. I would want to ask you what you think of the claims about inductive biases and difficulty of causal graph learning for transformers? A guess is that you could just add it on top of the base architecture as a MOA model with RL in it to solve some problems here but that feels like people from the larger labs might not realise that at first? 

Also, I wasn't only talking about GDL, there's like two or three other disciplines that also have some ways they believe that AGI will need other sorts of modelling capacity. 

Some of the organisation taking explicit bets from other directions are:

https://www.symbolica.ai/

https://www.verses.ai/genius

Symbolica is more on the same train as GDL but from a category theory perspective, the TL;DR of their take is that it takes other types of reasoning capacity in order to combine various data types into one model and that transformers aren't expressive nor flexible enough to support this.

For Verses, I think you should think ACS & Jan Kulveit Active Inference models & lack of planning with self in mind due to lacking information about what the self-other boundary is for auto-encoders compared to something that has an action-perception loop. 

I might write something up on this if you think it might be useful.

they can’t read Lesswrong or EA blogs

VPNs exist and are probably widely used in China + much of "all this work" is on ArXiv etc.

The initial distribution of values need not be highly related to the resultant values after moral philosophy and philosophical self-reflection. Optimizing hedonistic utilitariansm, for example, looks very little like any values from the outer optimization loop of natural selection.

At the start of the post I describe an argument I often hear:

But many people are under the misconception that the resulting “rat race”—the highly competitive and strenuous admissions ordeal—is the inevitable result of the limited class sizes among top schools and the strong talent in the applicant pools, and that it isn’t merely because of the reasons listed in (2). Some even go so far as to suggest that a better system would be to run a lottery for any applicant who meets a minimum “qualification” standard—under the assumption that there would be many such qualified students.

This is the argument that I'm responding to and refuting.

I've been thinking about a similar thing for myself https://vlad.roam.garden/Am-I-being-over-stimulated-by-audio-content and trying to scale down various uses of casual intellectual simulation audio content (did you know you can listen to things in the shower by using the waterproof earbuds 😅)

Although there would be pressure for an AI to not be exploitable, wouldn't there also be pressure for adaptability and dynamism? The ability to alter preferences and goals given new environments?

Thanks! I fixed the last paragraph accordingly. I indeed wanted to say faster-than-linearly for the highest feasible k.

My point wasn't about the duration of consciousness, but about the amount of lives that came into existence. Supposing some hundreds of millions of session starts per day, versus 400k human newborns, that's a lot more very brief AI lives than humans who will live "full" lives.

(Apparently we also have very different assumptions about the conversion rate between tokens of output and amount of consciousness experienced per second by humans, although I agree that most consciousness is not run inside AI slavery. But anyway that's another topic.) 

Thanks to Elizabeth for hosting me! I really enjoyed this conversation; "winning" is a concept that seems important and undervalued among rationalists, and I'm glad to have had the time to throw ideas around here.

I do feel like this podcast focused a bit more on some of the weirder or more controversial choices I made, which is totally fine; but if I were properly stating the case for "what is important about winning" from scratch, I'd instead pull examples like how YCombinator won, or how EA has been winning relative to rationality in recruiting smart young folks. AppliedDivinityStudies's "where are all the successful rationalists" is also great.

Very happy to answer questions ofc!

If a rock changed its behavior every time I video recorded/measured it, I actually would start to wonder about whether the rock, or something connected to it, had some awareness of my recording actions. However, that's just me, and I really happen to like rocks.