LessWrong 2.0 Reader

I was just thinking that anything is better than nothing. If I received the feedback you mentioned on some of my early downvoted posts, I'd have been less confused than I was.

the scope insensitive humans die and their society is extinguished

Ah, your reaction makes more sense given you think this is the proposal. But it's not the proposal. The proposal is that the scope-insensitive values flourish on Earth, and the scope-sensitive values flourish in the remaining cosmos.

As a toy example, imagine a distant planet with two species of alien: paperclip-maximisers and teacup-protectors. If you offer a lottery to the paperclip-maximisers, they will choose the lottery with the highest expected number of paperclips. If you offer a lottery to the teacup-satisfiers, they will choose the lottery with the highest chance of preserving their holy relic, which is a particular teacup.

The paperclip-maximisers and the teacup-protectors both own property on the planet. They negotiate the following deal: the paperclip-maximisers will colonise the cosmos, but leave the teacup-protectors a small sphere around their home planet (e.g. 100 light-years across). Moreover, the paperclip-maximisers promise not to do anything that risks their teacup, e.g. choosing a lottery that doubles the size of the universe with 60% chance and destroys the universe with 40% chance.

Do you have intuitions that the paperclip-maximisers are exploiting the teacup-protectors in this deal?

Do you think instead that the paperclip-maximisers should fill the universe with half paperclips and half teacups?

I think this scenario is a better analogy than the scenario with the drought. In the drought scenario, there is an object fact which the nearby villagers are ignorant of, and they would act differently if they knew this fact. But I don't think scope-sensitivity is a fact like "there will be a drought in 10 years". Rather, scope-sensitivity is a property of a utility function (or a value system, more generally).

You should make it totally clear which text is Nick Land's and which isn't. I spent like 10 minutes trying to figure it out when I first saw your post. 

It's not clear to me we'll have (or will "need") new paradigms before fully handing over all technical and strategic work to AIs which are capable enough to obsolete humans at all cognitive tasks.

If you want to not die to slop, then "fully handing over all technical and strategic work to AIs which are capable enough to obsolete humans at all cognitive tasks" not a thing which happens at all until the full superintelligence alignment problem is solved. That is how you die to slop.

If by superintelligence, you mean wildly superhuman AI, it remains non-obvious to me that new paradigms are needed (though I agree they will pretty likely arise prior to this point due to AIs doing vast quantity of research if nothing else). I think thoughtful and laborious implementation of current paradigm strategies (including substantial experimentation) could directly reduce risk from handing off to superintelligence down to perhaps 25% and I could imagine being argued considerably lower.

I find it hard to imagine such a thing being at all plausible. Are you imagining that jupiter brains will be running neural nets? That their internal calculations will all be differentiable? That they'll be using strings of human natural language internally? I'm having trouble coming up with any "alignment" technique of today which would plausibly generalize to far superintelligence. What are you picturing?

I would say it is basically-always true, but there are some fields (including deep learning today, for purposes of your comment) where the big hard central problems have already been solved, and therefore the many small pieces of progress on subproblems are all of what remains.

Maybe, but it is interesting to note that:

• A majority of productive work is occuring on small subproblems even if some previous paradigm change was required for this.
• For many fields, (e.g., deep learning) many people didn't recognize (and potentially still don't recognize!) that the big hard central problem was already solved. This potentially implies it might be non-obvious whether this has been solved and making bets on some existing paradigm which doesn't obviously suffice can be reasonable.

Things feel more continuous to me than your model suggests.

And insofar as there remains some problem which is simply not solvable within a certain paradigm, that is a "big hard central problem", and progress on the smaller subproblems of the current paradigm is unlikely by-default to generalize to whatever new paradigm solves that big hard central problem.

It doesn't seem clear to me this is true in AI safety at all, at least for non-worst-case AI safety.

I claim it is extremely obvious and very overdetermined that this will occur in AI safety sometime between now and superintelligence.

Yes, I added "prior to human obsolescence" (which is what I meant).

Depending on what you mean by "superintelligence", this isn't at all obvious to me. It's not clear to me we'll have (or will "need") new paradigms before fully handing over all technical and strategic work to AIs which are capable enough to obsolete humans at all cognitive tasks. Doing this hand over doesn't directly require understanding whether the AI is specifically producing alignment work that generalizes. For instance, the system might pursue routes other than alignment work and we might determine its judgement/taste/epistimics/etc are good enough based on examining things other than alignment research intended to generalize beyond the current paradigm.

If by superintelligence, you mean wildly superhuman AI, it remains non-obvious to me that new paradigms are needed (though I agree they will pretty likely arise prior to this point due to AIs doing vast quantity of research if nothing else). I think thoughtful and laborious implementation of current paradigm strategies (including substantial experimentation) could directly reduce risk from handing off to superintelligence down to perhaps 25% and I could imagine being argued considerably lower.

The version of this that OpenAI did [LW(p) · GW(p)] wasn't unexpected to me, but this one was, I wasn't sure about DeepMind. (Wait, now I'm remembering that "Google AI" is a separate group to "DeepMind"? Web says they were merged in 2023. I wonder which members on the inside this change reflects).

This post seems to assume that research fields have big hard central problems that are solved with some specific technique or paradigm.

This isn't always true. [...]

I would say it is basically-always true, but there are some fields (including deep learning today, for purposes of your comment) where the big hard central problems have already been solved, and therefore the many small pieces of progress on subproblems are all of what remains.

And insofar as there remains some problem which is simply not solvable within a certain paradigm, that is a "big hard central problem", and progress on the smaller subproblems of the current paradigm is unlikely by-default to generalize to whatever new paradigm solves that big hard central problem.

I agree that paradigm shifts can invalidate large amounts or prior work, but it isn't obvious whether this will occur in AI safety.

I claim it is extremely obvious and very overdetermined that this will occur in AI safety sometime between now and superintelligence. The question which you'd probably find more cruxy is not whether, but when - in particular, does it come before or after AI takes over most of the research?

... but (I claim) that shouldn't be the cruxy question, because we should not be imagining completely handing off the entire alignment-of-superintelligence problem to early transformative AI; that's a recipe for slop [LW · GW]. We ourselves need to understand a lot about how things will generalize beyond the current paradigm, in order to recognize when that early transformative AI is itself producing research which will generalize beyond the current paradigm, in the process of figuring out how to align superintelligence. If an AI assistant produces alignment research which looks good to a human user, but won't generalize across the paradigm shifts between here and superintelligence, then that's a very plausible way for us to die.

tell 12 to DM me i am so curious what exactly all this is about.

You might instead just directly study filmmaking.

Absolutely not. I cannot stress this enough.

Edit: I just saw your other comment that you studied filmmaking in college, so please excuse the over-explaining in this comment stuff that is no doubt oversimplified to you. Although I will state that there is no easier time to make films than in filmschool where classmates and other members of your cohort provide cast and crew, and the school provides facilities and equipment removing many of the logistical hurdles I enumerate.

So, (I mean this as an earnest question, not like a gotcha) why are you currently interested in general problem solving (as opposed to filmmaking?) Is it because general problem solving is intrinsically interesting/rewarding to you (if you could find a path to doing so?). Or because it just seemed pretty likely to be the a good step on your journey as a filmmaker? Or just because I gave a prompt to see if you could figure out a way to apply general problemsolving to your life, and there was at least some appeal to that?

More so the last one, I'm bad at general problem solving, I'm also very messy and disorganized because I can't find the right "place" for things which suggests I'm very bad at predicting my own future self in such a way that I can place objects (and notes for that matter) in assigned spaces which will be easy and obvious for me to recall later.

That being said my only interest, my single minded terminal goal is to tell good visual stories but to quote Orson Welles "filmmaking is 2% filmmaking 98% hustling". I'm not a hustler. The logistical and financial problem solving that facilitate the storytelling/filmmaking are things I am absolutely terrible at. So much of filmmaking is figuring out logistics, time management, practical problem solving that has little or nothing to do with the aesthetic intentions. The other half is the sociological component but that seems less relevant to metacgonition.

A poet friend of mine describes the tremendous difference between when she wants to create - she picks up a pen and paper. And a filmmaker who needs to move heaven and earth.

Music videos in fact simplify a lot of the logistical problems of filmmaking because they are shorter, there's less of an onus to persuade and pitch an idea, since the band already are invested emotionally (and financially) in having a video made. You just need to help them get their story across, not sell them on your own story. However that still requires getting commisions, marketing, and presents it's own logistical challenges owing to shorter turnarounds.

The simple fact is I'm not a schmoozer or a networker - whether you want to make films or music videos, you need someone to give you the opportunity (usually that means finances, but not necessarily). That's the first hurdle. The second hurdle is that you can have a great idea for a music video, can storyboard it, it can all make sense in aesthetic terms but the logistics of making it happen are another thing entirely. You can have something that makes sense as story, but making it requires broad problem solving skills... more so when you don't have finances.

Now assuming that a musician or band does commission me for a music video, they've agreed to a pitch, which happens with more and more frequency as my reputation has grown over 5 years of doing this - now what?

Firstly you need a space to film this music video. Then you need to consider, with musicians you often need to find a time when they can all take time off of work and doesn't impinge on their music-making. Now you find yourself trying to contort the logistics into a window of time that allows you to bump in and out of several locations, set up camera, lights, change costumes and makeup, maintain continuity (although less so an issue in music videos). I find myself writing gantt charts and estimating "turnarounds" and finding the most expedient order to put things in.

The space to film needs to be appropriate aesthetically, it needs to add to the story, the larger the better. It needs the right lighting, that involves a whole host of considerations beyond the aesthetics of lighting and colour theory like - how many watts can we draw from the wall? If we want a diffuse light, where do we physically put the sheet or diffuser in a confined but aesthetically appropriate space? What if we're not allowed to move certain furnishings as part of the deal with the owners of the space but it's really ruining our shot? How do we solve that?

I could go on and on and on. Do you know how many film shoots I've been on where police were called? The storytelling, the shot selection, the colour palettes, the communication of gesture and intent to performers, the editing and selection of shots, the rhythm and pacing... that's not the hard part: money and logistics are.

Many of these problems could be solved (re: outsourced) with more finances, being able to hire other people who specialize in those things. Most people say "you should get a producer" and it's like... yeah, how do I find this magical person?

When I have a great story in my head, and you ask me "how do you do that?" - i shrug. I don't know.