LessWrong 2.0 Reader

This strongly updates me towards expecting the institute to produce useful work.

[note: I initially read this post like a month ago, forgot to comment, have not reread now before commenting]

Thanks for writing this, as the creative lead I really like seeing what people think!

I like your geological review of Song Of The Artesian Water. (I also like your factual nitpick of Bold Orion, elsewhere.)

I want to somewhat disagree with some of your overall approach to the content of Solstice, largely exemplified here:

There is a phenomenon in comparative theology where people are much more sensitive to whether the theology is correct in a talk than in a song. If you say a controversial doctrine in a talk, people might get upset, come up and argue with you afterwards, or even start attending a different congregation. If you put the same doctrine in a song, people are more likely to smile and sing along. People know this, and sometimes write extremely partisan songs that take advantage of the phenomenon. 'Know This That Every Soul Is Free' comes to mind as an example. The appropriate response is to take the content of songs as seriously as the content of talks. Their words probably do reflect the author's intent.

This song is the most explicit statement of the future that the rationalist community hopes to build.

A thing about Solstice is that while there is a creative lead who curates and sometimes edits the content, plots the thematic arc, and writes new content to connect the dots and lay out the overall ~philosophy and claims of the program (which might be different year to year!), the content is really quite crowdsourced and represents a patchwork of beliefs and ideals that all point in somewhat the same direction but not exactly the same direction.

As I said at the start - it's not a liturgy, it's a reflection. (A series of reflections, rather.)

Even though I chose every piece in the program, edited some of the song lyrics, and had creative input into the speeches, I would say the only pieces in the program that I agree with fully with no reservations are the speeches/interludes I wrote myself. (Maybe not even those depending how strict you get with "no reservations" - there are any number of clarifications I could add that would slightly improve the accuracy of my content but make the program interminably long and boring. (I spent a lot of editing time cutting wordcount.))

With speeches other people wrote for the program, I made a lot of suggestions; the authors took some of my suggestions and not others. This is as it should be! A lot of the point of having community members get up and speak their own words is so that they can speak from the heart, from what they know and believe, from their own experiences in the world. Of course I'm not going to choose a speaker who wants to talk about something antithetical or just orthogonal to my Solstice concept - but neither do I expect anyone to align 100% with all of my beliefs and priorities, and this is fine and good.

With songs it's even worse. Songs have to rhyme and scan. This trades off against optimizing your meaning with anywhere near as much precision as you can in prose. Obviously this doesn't mean you can throw out the whole project of having your words mean things you endorse! But you probably won't be able to chisel your meaning as finely; you have to accept some degree of poetic license and/or being merely directionally correct.

One example that stuck in my mind is that - admittedly in a footnote to your other post - you said:

The most doomy statement I recall implied that there was a proof that alignment would fail. The speaker expressed hope that there might be a subtle flaw in that proof, much like there was a subtle flaw in Kelvin's argument that manned powered flight would not work."

This refers to these lines in I Have Seen The Tops Of Clouds:

This isn't a thing that our past selves expected -
Lord Kelvin assured us that steel cannot fly;
His mistake was quite subtle, and all we need hope for
Is similar errors in proofs we'll all die.

I don't know if Dan Speyer (who wrote these lines) believes it is literally true that there is a literal proof we'll all die. I chose this song and wrote new music for it because I felt strongly that the song said an important thing that I wanted in my program, and I certainly don't literally believe this. I think this is Close Enough to things I do believe that I don't feel it's out of place in my program. (And there are other parts of the song that I am much more closely aligned with and which are the main reasons I wanted to include it.)

Editing songs is quite costly. Firstly in time and effort (which are at an extreme premium during Solstice prep, it's SO much work) - making things rhyme and scan is no small task. But also, many Solstice songs are passed along from Solstice to Solstice with the expectation that, over time, people will learn them and sing them from memory. (This expectation is very much a reality with a lot of Solstice songs at this point.) Violating people's expectations about a song they already know is a pretty significant cost actually - not insurmountable, and we do edit songs sometimes, but not at all trivial.

(I made a tiny change in What A Wonderful World - from "they'll learn much more than I'll ever know" to "they MAY learn much more than I'll ever know" - and I got a LOT of feedback about that!)

Not to mention that you'll never get all rationalists to agree on what the One Truest Possible Form Of The Song With No Errors Or Wrong Emphases Or Unwanted Implications is, and we don't have a central authority to dictate that, which is good. (Another, related, part of your approach I disagreed with is the note that maybe Gretta shouldn't be sharing wisdom on a problem she hasn't fully solved for herself. I don't think anyone has fully solved the problem of grief and fear for themself, and I don't think anyone ought to go around claiming they have. We're all just people trying to figure out the world and sharing our little bits of hard-won wisdom with each other, trying to piece together a patchwork model of the world and how to live in it.)

With Great Transhumanist Future, I very specifically tried to frame the song as one possible vision of a bright future, rather than the one true thing we're all aiming at. This is very much a case where I endorse the song as part of a pluralistic mosaic of visions, and agree with many parts of it, but don't necessarily see every thing it yearns for as a thing I myself want in my future, and absolutely don't want to claim anyone else must. But I think the rich detail of a specific vision is far more interesting and motivating (including for noticing where one might disagree!) than a sanded-down universal vision that only contains what everyone agrees on.

It's fair that there's only so much framing can do, though, and like, it makes a lot of sense to me that given that you feel in general only very partially value- and goal-aligned with rationalists, you will also feel pretty unaligned with a lot of the stuff in Solstice and largely with the arc overall. This is very valid, and I appreciate you sharing this perspective! I just wanted to note that I think you might be interpreting the program in a significantly more narrow and prescriptive way than it was intended.

(You might be interested in the Dialogue I already did [LW · GW] with habryka.)

Thanks! Added.

Huh, that looks like it had a persistent effect too. Looks to me like you're a lot more productive when you work on your own stuff, now.

This seems quite promising to me. My primary concern is that I feel like NIST is really not an institution well-suited to house alignment research. 

My current understanding is that NIST is generally a very conservative organization, with the primary remit of establishing standards in a variety of industries and sciences. These standards are almost always about things that are extremely objective and very well-established science. 

In contrast, AI Alignment seems to me to continue to be in a highly pre-paradigmatic state, and the standards that I can imagine us developing there seem to be qualitatively very different from the other standards that NIST has historically been in charge of developing. It seems to me that NIST is not a good vehicle for the kind of cutting edge research and extremely uncertain environment in which things like alignment and evals research have to happen. 

Maybe other people have a different model of different US governmental institutions, but at least to me NIST seems like a bad fit for the kind of work that I expect Paul to be doing there. 

Thanks! Added.

Oh, I have indeed used this to update that OpenAI deeply misunderstands alignment, and this IMO has allowed me to make many accurate predictions about what OpenAI has been doing over the last few years, so I feel good about interpreting it that way.

It's a fair point that wisdom might not be straightforwardly safety-increasing. If someone wanted to explore e.g. assumptions/circumstances under which it is vs isn't, that would certainly be within scope for the competition.

Thanks! Added.

Relevant note from the entry:

Me: I was hesitant to add a lecture series to this list at first. I changed my mind after listening to the first video, where Meehl provides interesting details (gossip, almost) about the life of an academic and the various personalities of his successful academic peers.