LessWrong 2.0 Reader

Useful comparison; but I'd say AI is better compared to biology than to computer security [LW · GW] at the moment.

Oh this is a great way of laying it out. Agreed on many points, and I think this may have made some things easier for me to see, likely some of that is actual update that changes opinions I've shared before that you're disagreeing with. I'll have to ponder.

Fantastic, thanks!

I do think that if you found a zero-RL path to the same (or better) endpoint, it would often imply that you've grasped something about the problem more deeply, and that would often imply greater safety.

Some applications of RL are also just worse than equivalent options. As a trivial example, using reward sampling to construct a gradient to match a supervised loss gradient is adding a bunch of clearly-pointless intermediate steps.

I suspect there are less trivial cases, like how a decision transformer isn't just learning an optimal policy for its dataset but rather a supertask: what different levels of performance look like on that task. By subsuming an RL-ish task in prediction, the predictor can/must develop a broader understanding of the task, and that understanding can interact with other parts of the greater model. While I can't currently point to strong empirical evidence here, my intuition would be that certain kinds of behavioral collapse would be avoided by the RL-via-predictor because the distribution is far more explicitly maintained during training.^[1][2]

But there are often reasons why the more-RL-shaped thing is currently being used. It's not always trivial to swap over to something with some potential theoretical benefits when training at scale. So long as the RL-ish stuff fits within some reasonable bounds, I'm pretty okay with it and would treat it as a sufficiently low probability threat that you would want to be very careful about how you replaced it, because the alternative might be sneakily worse.

1. ^{^}

   KL divergence penalties [LW · GW] are one thing, but it's hard to do better than the loss directly forcing adherence to the distribution.

2. ^{^}

   You can also make a far more direct argument about model-level goal agnosticism [LW · GW] in the context of prediction.

There's so much discussion, in safety and elsewhere, around the unpredictability of AI systems on OOD inputs. But I'm not sure what that even means in the case of language models.

With an image classifier it's straightforward. If you train it on a bunch of pictures of different dog breeds, then when you show it a picture of a cat it's not going to be able to tell you what it is. Or if you've trained a model to approximate an arbitrary function for values of x > 0, then if you give it input < 0 it won't know what to do.

But what would that even be with an LLM? You obviously (unless you're Matt Watkins) can't show it tokens it hasn't seen, so 'OOD' would have to be about particular strings of tokens. It can't be simply about strings of tokens it hasn't seen, because I can give it a string I'm reasonably confident it hasn't seen and it will behave reasonably, eg:

Define a fnurzle as an object which is pink and round and made of glass and noisy and 2.5 inches in diameter and corrugated and sparkly. If I'm standing in my living room and holding a fnurzle in my hand and then let it go, what will happen to it?

…In summary, if you let go of the fnurzle in your living room, it would likely shatter upon impact with the floor, possibly emitting noise, and its broken pieces might scatter or roll depending on the surface.

(if you're not confident that's a unique string, add further descriptive phrases to taste)

So what, exactly, is OOD for an LLM? I…suppose we could talk about the n-dimensional shape described by the points in latent space corresponding to every input it's seen? That feels kind of forced, and it's certainly not obvious what inputs would be OOD. I suppose eg 1700 repetitions of the word 'transom' followed by a question mark would seem intuitively OOD? Or the sorts of weird adversarial suffixes found in eg Lapid et al (like 'équipesmapweiábardoMockreas »,broughtDB multiplicationmy avo capsPat analysis' for Llama-7b-chat) certainly seem intuitively OOD. But what about ordinary language -- is it ever OOD? The issue seems vexed.

Yep, lots of people independently complain about "lab." Some of those people want me to use scary words in other places too, like replacing "diffusion" with "proliferation." I wouldn't do that, and don't replace "lab" with "mega-corp" or "juggernaut," because it seems [incorrect / misleading / low-integrity].

I'm sympathetic to the complaint that "lab" is misleading. (And I do use "company" rather than "lab" occasionally, e.g. in the header.) But my friends usually talk about "the labs," not "the companies." But to most audiences "company" is more accurate.

I currently think "company" is about as good as "lab." I may change the term throughout the site at some point.

I wouldn't call this "AI lab watch." "Lab" has the connotation that these are small projects instead of multibillion dollar corporate behemoths.

This seems like a good point. Here's a quick babble of alts (folks could react with a thumbs-up on ones that they think are good).

AI Corporation Watch | AI Mega-Corp Watch | AI Company Watch | AI Industry Watch | AI Firm Watch | AI Behemoth Watch | AI Colossus Watch | AI Juggernaut Watch | AI Future Watch

I currently think "AI Corporation Watch" is more accurate. "Labs" feels like a research team, but I think these orgs are far far far more influenced by market forces than is suggested by "lab", and "corporation" communicates that. I also think the goal here is not to point to all companies that do anything with AI (e.g. midjourney) but to focus on the few massive orgs that are having the most influence on the path and standards of the industry, and to my eye "corporation" has that association more than "company". Definitely not sure though.

What's the chance of a 2nd LessOnline?

Um, one part of me is (as is not uncommon) really believes in [LW · GW] this event and thinks it's going to be the best effort investments Lightcones' ever made (though this part of me currently has one or two other projects and ideas that it believes in maybe even more strongly), that's part of me is like "yeah this should absolutely happen every year", though as I say I get this feeling often about projects that often end up looking different to how I dreamed them when they finally show up in reality. I think that part would feel validated by the event turning out to be awesome and people finding it was worthwhile to come. Then there's the question of how much resources Lightcone actually has and whether we'll successfully fundraise and whether this will be one of the few projects we're investing a few staff-months in a year from now. I think my probabilities just went from 80% to 50% to 99% to... 30%. Overall it depends on how good this event is, which varies on a log scale.

I think there are worlds where we do it again and invest less effort into it, also there's worlds where we do it again and invest more effort into it. I think there's also a bunch of worlds where we're happy about this event but try a subtly different one next time (e.g. me and a teammate generated like 5 other serious event contenders before this one, including things more like workshops or academic conferences than like large festivals, and perhaps we'll try a different thing next). I think I like that this event is essentially open-invite and trying to be more big-tent, and I hope to do more things like this, so that even if we change what sorts of events we run anyone will just be able to buy a ticket. There's also worlds where we stop exploring having an events team for our campus and stop running events.

As one datapoint, in 2021 I organized a 60-person private event called the Sanity & Survival Summit for rationalist and folks working professionally on x-risk stuff, and I thought we'd maybe make that a yearly thing, and a year later we sort of last-minute/impromptu ran another version of it called Palmcone (it was in the Bahamas) for 80-100 people, and then we made the Lightcone Offices to try and get a more permanent version of the EA/x-risk things in the Bay, and then we scrapped the whole thing as we uninvested in the professional x-risk/EA ecosystem. That's a possible trajectory things could take.

That just sounds great, thanks.

That's unfortunate that you are less likely to come, and I'm glad to get the feedback. I could primarily reply with reasons why I think it was the right call (e.g. helpful for getting the event off the ground, helpful for pinpointing the sort of ideas+writing the event is celebrating, I think it's prosocial for me to be open about info like this generally, etc) but I don't think that engages with the fact that it left you personally less likely to come. I still overall think if the event sounds like a good time to you (e.g. interesting conversations with people you'd like to talk to and/or exciting activities) and it's worth the cost to you then I hope you come :-)

Maybe to clarify my comment: I was merely describing my (non-endorsed^[1]) observed emotional content wrt the festival, and my intention with thw comment was not to wag my finger at you guys in the manner of "you didn't invite me".

I wonder whether other people have a similar emotional reaction.

I appreciate Lightcone being open with the information around free invitations though! I think I'd have bought a ticket anyway${}_{65\%}$ if I had time around that weekend, and I think I'd probably have a blast if I would attend.

Btw: What's the chance of a 2nd LessOnline?