LessWrong 2.0 Reader

Is there another US governmental organization that you think would be better suited? My relatively uninformed sense is that there's no real USFG organization that would be very well-suited for this—and given that NIST is probably one of the better choices.

Going to the moon

Say you’re really, really worried about humans going to the moon. Don’t ask why, but you view it as an existential catastrophe. And you notice people building bigger and bigger airplanes, and warn that one day, someone will build an airplane that’s so big, and so fast, that it veers off course and lands on the moon, spelling doom. Some argue that going to the moon takes intentionality. That you can’t accidentally create something capable of going to the moon. But you say “Look at how big those planes are getting! We've gone from small fighter planes, to bombers, to jets in a short amount of time. We’re on a double exponential of plane tech, and it's just a matter of time before one of them will land on the moon!”

Contra Scheming AIs

There is a lot of attention on mesaoptimizers, deceptive alignment, and inner misalignment. I think a lot of this can fall under the umbrella of "scheming AIs". AIs that either become dangerous during training and escape, or else play nice until humans make the mistake of deploying them. Many have spoken about the lack of an indication that there's a "humanculi-in-a-box", and this is usually met with arguments that we wouldn't see such things manifest until AIs are at a certain level of capability, and at that point, it might be too late, making comparisons to owl eggs, or baby dragons. My perception is that getting something like a "scheming AI" or "humanculi-a-box" isn't impossible, and we could (and might) develop the means to do so in the future, but that it's a very, very different kind of thing than current models (even at superhuman level), and that it would take a degree of intentionality.

As someone who likes transcribing songs,

1) I endorse the above

2) if you ask me to transcribe a song I will often say yes (if it's not very frequent) (it costs time but not that much cognitive work for me so I experience reasonable amounts of this as fun)

One thing that makes this hard to automate is human imprecision in generating a recording, espeically with rhythm: notes encode frequencies but also timings and durations, and humans performing a song will never get those things exactly precise (nor should they - good performance tends to involve being a little free with rhythms in ways that shouldn't be directly reflected in the sheet music), so any automatic transcriber will get silly-looking slightly off rhythms that still need judgment to adjust.

Couple of thoughts re: NIST as an institution:

1. FWIW NIST, both at Gaithersburg and Boulder, is very well-regarded in AMO physics and slices of condensed matter. Bill Phillips (https://www.nist.gov/people/william-d-phillips) is one big name there, but AIUI they do a whole bunch of cool stuff. Which doesn't say much one way or the other about how the AI Safety Institute will go! But NIST has a record of producing, or at least sheltering, very good work in the general vicinity of their remit.
2. Don't knock metrology! It's really, really hard, subtle, and creative, and you all of a sudden have to worry about things nobody has ever thought about before! I'm a condensed matter theory guy, more or less, but I go to metrology talks every once in a while and have my mind blown.

Oh, also I wanted to comment on the section of your other post [LW · GW] where you mention that Solstice contained a number of what felt like barbs at religion.

This is a fairly valid complaint, I think. I am sorry you felt barbed. I agree that barbs at the outgroup are not a great Solstice strategy; I specifically aimed to keep conflict-theoretic content out of my program (and edited e.g. some of the Underrated Reasons To Be Thankful accordingly). I think these were less salient to me for basically cultural and positional reasons, and it makes sense they were much more salient to you.

It's also the case that there's not much I would do differently even knowing in advance that someone would have this reaction, because (a) I disagree with some of the examples, (b) nearly all of the examples are in songs, written by people other than me, and already known and beloved within the community, and as such very difficult to change.

(The one example that's not from a song I think you may have misheard - what I said there was "this next song started as a sort of, intra-religious rebuttal against overly literalist interpretations of the Bible", which is not anti-religious.)

I guess I also draw a distinction between rejecting some religious practices and being mean to religious people.

The Brighter Than Today verse is very much a thing I wouldn't write that way myself but will by no means change because everybody is extremely attached to it. (I've heard a very similar complaint about it from a very secular friend, and I think you and she are basically right but I don't disagree with the song strongly enough to refuse to sing it as is.)

But like I do think there is pretty substantial validity to your feeling here anyway, especially given that it is not uncommon for rationalists to be much more antitheist and anti-religious-people than this. Sorry about that.

Slavery is one subject that it's highly likely ChatGPT is specifically programmed to handle differently for political reasons. How did you get around this problem?

We view it as an interesting open question whether it is possible to develop SSM-like models with greater expressivity for state tracking that also have strong parallelizability and learning dynamics

Surely fundamentally at odds? You can't spend a while thinking without spending a while thinking.

Of course, the lunch still might be very cheap by only spending a while thinking a fraction of the time or whatever.

This strongly updates me towards expecting the institute to produce useful work.

[note: I initially read this post like a month ago, forgot to comment, have not reread now before commenting]

Thanks for writing this, as the creative lead I really like seeing what people think!

I like your geological review of Song Of The Artesian Water. (I also like your factual nitpick of Bold Orion, elsewhere.)

I want to somewhat disagree with some of your overall approach to the content of Solstice, largely exemplified here:

There is a phenomenon in comparative theology where people are much more sensitive to whether the theology is correct in a talk than in a song. If you say a controversial doctrine in a talk, people might get upset, come up and argue with you afterwards, or even start attending a different congregation. If you put the same doctrine in a song, people are more likely to smile and sing along. People know this, and sometimes write extremely partisan songs that take advantage of the phenomenon. 'Know This That Every Soul Is Free' comes to mind as an example. The appropriate response is to take the content of songs as seriously as the content of talks. Their words probably do reflect the author's intent.

This song is the most explicit statement of the future that the rationalist community hopes to build.

A thing about Solstice is that while there is a creative lead who curates and sometimes edits the content, plots the thematic arc, and writes new content to connect the dots and lay out the overall ~philosophy and claims of the program (which might be different year to year!), the content is really quite crowdsourced and represents a patchwork of beliefs and ideals that all point in somewhat the same direction but not exactly the same direction.

As I said at the start - it's not a liturgy, it's a reflection. (A series of reflections, rather.)

Even though I chose every piece in the program, edited some of the song lyrics, and had creative input into the speeches, I would say the only pieces in the program that I agree with fully with no reservations are the speeches/interludes I wrote myself. (Maybe not even those depending how strict you get with "no reservations" - there are any number of clarifications I could add that would slightly improve the accuracy of my content but make the program interminably long and boring. (I spent a lot of editing time cutting wordcount.))

With speeches other people wrote for the program, I made a lot of suggestions; the authors took some of my suggestions and not others. This is as it should be! A lot of the point of having community members get up and speak their own words is so that they can speak from the heart, from what they know and believe, from their own experiences in the world. Of course I'm not going to choose a speaker who wants to talk about something antithetical or just orthogonal to my Solstice concept - but neither do I expect anyone to align 100% with all of my beliefs and priorities, and this is fine and good.

With songs it's even worse. Songs have to rhyme and scan. This trades off against optimizing your meaning with anywhere near as much precision as you can in prose. Obviously this doesn't mean you can throw out the whole project of having your words mean things you endorse! But you probably won't be able to chisel your meaning as finely; you have to accept some degree of poetic license and/or being merely directionally correct.

One example that stuck in my mind is that - admittedly in a footnote to your other post - you said:

The most doomy statement I recall implied that there was a proof that alignment would fail. The speaker expressed hope that there might be a subtle flaw in that proof, much like there was a subtle flaw in Kelvin's argument that manned powered flight would not work."

This refers to these lines in I Have Seen The Tops Of Clouds:

This isn't a thing that our past selves expected -
Lord Kelvin assured us that steel cannot fly;
His mistake was quite subtle, and all we need hope for
Is similar errors in proofs we'll all die.

I don't know if Dan Speyer (who wrote these lines) believes it is literally true that there is a literal proof we'll all die. I chose this song and wrote new music for it because I felt strongly that the song said an important thing that I wanted in my program, and I certainly don't literally believe this. I think this is Close Enough to things I do believe that I don't feel it's out of place in my program. (And there are other parts of the song that I am much more closely aligned with and which are the main reasons I wanted to include it.)

Editing songs is quite costly. Firstly in time and effort (which are at an extreme premium during Solstice prep, it's SO much work) - making things rhyme and scan is no small task. But also, many Solstice songs are passed along from Solstice to Solstice with the expectation that, over time, people will learn them and sing them from memory. (This expectation is very much a reality with a lot of Solstice songs at this point.) Violating people's expectations about a song they already know is a pretty significant cost actually - not insurmountable, and we do edit songs sometimes, but not at all trivial.

(I made a tiny change in What A Wonderful World - from "they'll learn much more than I'll ever know" to "they MAY learn much more than I'll ever know" - and I got a LOT of feedback about that!)

Not to mention that you'll never get all rationalists to agree on what the One Truest Possible Form Of The Song With No Errors Or Wrong Emphases Or Unwanted Implications is, and we don't have a central authority to dictate that, which is good. (Another, related, part of your approach I disagreed with is the note that maybe Gretta shouldn't be sharing wisdom on a problem she hasn't fully solved for herself. I don't think anyone has fully solved the problem of grief and fear for themself, and I don't think anyone ought to go around claiming they have. We're all just people trying to figure out the world and sharing our little bits of hard-won wisdom with each other, trying to piece together a patchwork model of the world and how to live in it.)

With Great Transhumanist Future, I very specifically tried to frame the song as one possible vision of a bright future, rather than the one true thing we're all aiming at. This is very much a case where I endorse the song as part of a pluralistic mosaic of visions, and agree with many parts of it, but don't necessarily see every thing it yearns for as a thing I myself want in my future, and absolutely don't want to claim anyone else must. But I think the rich detail of a specific vision is far more interesting and motivating (including for noticing where one might disagree!) than a sanded-down universal vision that only contains what everyone agrees on.

It's fair that there's only so much framing can do, though, and like, it makes a lot of sense to me that given that you feel in general only very partially value- and goal-aligned with rationalists, you will also feel pretty unaligned with a lot of the stuff in Solstice and largely with the arc overall. This is very valid, and I appreciate you sharing this perspective! I just wanted to note that I think you might be interpreting the program in a significantly more narrow and prescriptive way than it was intended.