LessWrong 2.0 Reader

We're looking for speakers for the Boston Solstice. This year Solstice is December 28th, 7pm. Being a speaker at solstice is pretty straightforward; public speaking skill is useful but you can read off a script, don't feel like you need to memorize something.

If you're at all interested, reach out. We have speeches ranging from very short and silly to a couple of pages and somber.

Additionally, if you feel like you have an original speech on the themes of persistence or camaraderie, especially if you feel you have a good speech about not giving up even when it's hard, then please feel free to send a draft! The overall arc is set at this point but you might have something better for a given slot.

I agree that those are the thoughts at the surface-level of Bryce in those situations, and they are not the same as "it's wrong/stupid to enjoy eating ice cream."

But I think in many cases, they often do imply "and you are stupid/irrational if knowing these things does not spoil your enjoyment or shift your hedonic attractor." And even if Bryce genuinely doesn't feel that way, I hope they would still be very careful with their wording to avoid that implication.

Tentative support for only auto-importing the first few paragraphs, if not that then start by auto-importing the whole post and waiting until anybody complains. My guess (~65%?) is that somebody will. Against having an LLM extract some important highlights- if doing highlights is the way to go I think whoever nominated the piece for the review can find the highlights?

I'd love it if I could use LessWrong as a central place to read rationalsphere content, and since more and more rationalist sphere writers are writing elsewhere this seems like it's worth trying.

Great post! I especially agree that for most independent researchers, applying to PHDs before you necessarily want one would be a helpful option to have as a backstop for if your near term career plans don't work out - and people should apply early because there's such a long lag time between application and starting.

I think it's also worth emphasising that if you have a non-standard work history (or are a bit junior), but might want to work in the United States, pursuing higher education in the US is one of the easiest ways to secure long-term work authorisation (And if someone funds your PhD, is radically cheaper than almost every alternative)

I know the causal closure of the physical as the principle that nothing non-physical influences physical stuff, so that would be the causal closure of the bottom level of description (since there is no level below the physical), rather than the upper.

So if you mean by that that it's enough to simulate neurons rather than individual atoms, that wouldn't be "causal closure" as Wikipedia calls it.

Yeah the LW team has been doing this sort of thing internally, still in the experimental phase. I don't know if we've used all the tricks listed here yet. 

For this year's LessWrong Review, we're building UI to make it much easier to import linkposts from other blogs, since a lot of important rationalsphere or AI Safety content lives in other places, and backdate it such that it's eligible for the Review.

It's actually pretty easy to automatically import all the text from a url in most cases (We're looking into auto-importing PDFs of papers, which I suspect is doable but haven't checked), and in many cases I think this would basically be preferred, but it's also kinda exploitable in ways I don't know that I'd endorse. (i.e. some authors are probably happy to have people crosspost stuff while nominating it for Best of LessWrong, other authors might feel violated)

Three options are:

• only auto-import the first few paragraphs, ending with a load more
• have an LLM extract some important highlights. (I'm ignoring "have an LLM summarize it" because they suck at that, but I think they're decent at identifying key paragraphs)
• start off by auto-importing the whole post, and then wait until anybody complains.

I'd probably be limiting this to users who are otherwise eligible to nominate (i.e. their account is at least two years old, and maybe they have like 100 karma), so randos can't go crazy with it. Admins will be seeing all posts imported this way so we can be sanity checking things.

Curious what people think.

I think one clear aspect of the stories here, yours and John's, relates to what I'll call asymmetric information flows. Basically, the times at which the information, that no one is trying to keep secret, become known to the relevant parties.

Of course understanding what a good update frequency is for various situations should be is a tricky thing itself.

Agreed that scaling rather than addition is usually the better way to think about probabilities. In this case we've done so little work on alignment that I think it might actually be more like additive, from 1% to 26% or 50% to 75% with ten extra years relative to the real current odds if we press ahead - which nobody knows.

I'm pretty sure it would be an error to trust anyone's estimate at this time, because people with roughly equal expertise and wisdom (e.g., Yudkowsky and Christiano) give such wildly different odds. And the discussions between those viewpoints always trail off into differing intuitions.

I also give alignment by default very poor odds, and prosaic alignment as it's usually discussed. But there are some pretty obvious techniques that are so low-tax that I think they'll be implemented even by orgs that don't take safety very seriously.

I'm curious if you've read my Instruction-following AGI is easier and more likely than value aligned AGI [LW · GW] and/or Internal independent review for language model agent alignment [AF · GW] posts. Instruction-following is human-in-the-loop so that may already be what you're referring to. But some of the techniques in the independent review post (which is also a review of multiple methods) go beyond prosaic alignment to apply specifically to foundation model agents. And wisely-used instruction-following gives corrigibility with a flexible level of oversight.

I'm curious what you think about those techniques if you've got time to look.

I think public acceptance of a pause is only part of the issue. The Chinese might actually not pursue AGI if they didn't have to race the US. But Russia and North Korea will most certainly pursue it (although they've got very limited resources and technical chops to make lots of progress in new foundation models, but they still might get to real AGI based on turning next-gen (which there's not time to pause) foundation models into scaffolded cognitive architectures.

But yes, I do think there's a chance we could get the US and European public to support a pause using some of the framings you suggest. But we'd better be sure that's a good idea. Lots of people, notably Russians and North Koreans, are genuinely way less cautious even than Americans - and absolutely will not honor agreements to pause.

Those are some specifics; in general I think it's only useful to talk about what "we" "should" do in the context of what particular actors actually are likely to do in different scenarios. Humanity is far from aligned, and that's a problem.

LLM support for writing LessWrong posts: virtual comments.

Back in August I discussed with Ruby & Oliver a bit about how to integrate LLMs into LW2 in ways which aren't awful, particularly using the new 'prompt caching' feature. To summarize one idea: we can use long-context LLMs with prompt caching to try to simulate various LW users of diverse perspectives to write useful feedback on drafts for authors.

(Prompt caching (eg) is the Transformer version of the old RNN hidden-state caching trick, where you run an input through the (deterministic) NN, and then save the intermediate version, and apply that to arbitrarily many future inputs, to avoid recomputing the first input each time, which is the naive way to do it. You can think of it as a lightweight finetuning. This is particularly useful if you are thinking about having large generic prompts---such as an entire corpus. A context window of millions of tokens might take up to a minute & $1 to compute currently, so you definitely need to be careful and don't want to compute more than once.)

One idea would be to try to use LLMs to offer feedback on drafts or articles. Given that tuned LLM feedback from Claude or ChatGPT is still not that great, tending towards sycophancy or obviousness or ChatGPTese, it is hardly worthwhile running a post through a generic "criticize this essay" prompt. (If anyone on LW2 wanted to do such a thing, they are surely capable of doing it themselves, and integrating it into LW2 isn't that useful. Removing the friction might be helpful, but it doesn't seem like it would move any needles.)

So, one way to force out more interesting feedback would be to try to force LLMs out of the chatbot assistant mode-collapse, and into more interesting simulations for feedback. There has been some success with just suggestively-named personas or characters in dialogues (you could imagine here we'd have "Skeptic" or "Optimist" characters), but we can do better. Since this is for LW2, we have an obvious solution: simulate LW users! We know that LW is in the training corpus of almost all LLMs and that writers on it (like myself) are well-known to LLMs (eg. truesight). So we can ask for feedback from simulated LWers: eg. Eliezer Yudkowsky or myself or Paul Christiano or the author or...

This could be done nicely by finetuning a "LW LLM" on all the articles & comments, with associated metadata like karma, and then feeding in any new draft or article into it, and sampling a comment from each persona. If there is some obvious criticism or comment Eliezer Yudkowsky would make on a post, which even a LLM can predict, why not deal with it upfront instead of waiting for the real Eliezer to comment (which is also unlikely to ever happen these days)? And one can of course sample an entire comment tree of responses to a 'virtual comment', with the LLM predicting the logical respondents.

This can further incorporate the draft's author's full history, which will usually fit into a multi-million token context window. So their previous comments and discussions, full of relevant material, will get included. This prompt can be cached, and used to sample a bunch of comment-trees. (And if finetuning is infeasible, one can try instead to put the LW corpus into the context and prompt-cache that before adding in the author's corpus.)

The default prompt would be to prompt for high-karma responses. This might not work, because it might be too hard to generate good high-quality responses blindly in a feedforward fashion, without any kind of search or filtering. So the formatting of the data might be to put the metadata after a comment, for ranking purposes: so the LLM generates a response and only then a karma score, and then when we sample, we simply throw out predicted-low-score comments rather than waste the author's time looking at them. (When it comes to these sorts of assistants, I strongly believe 'quality > quantity', and 'silence is golden'. Better to waste some API bills than author time.)

One can also target comments to specific kinds of feedback, to structure it better than a grab-bag of whatever the LLM happens to sample. It would be good to have (in descending order of how likely to be useful to the author) a 'typo' tree, a 'copyediting'/'style'/'tone' tree, 'confusing part', 'terminology', 'related work', 'criticism', 'implications and extrapolations', 'abstract/summary' (I know people hate writing those)... What else? (These are not natural LW comments, but you can easily see how to prompt for them with prompts like "$USER $KARMA $DATE | Typo: ", etc.)

As they are just standard LW comments, they can be attached to the post or draft like regular comments (is this possible? I'd think so, just transclude the comment-tree into the corresponding draft page) and responded to or voted on etc. (Downvoted comments can be fed back into the finetuning with low karma to discourage feedback like that.) Presumably at this point, it would not be hard to make it interactive, and allow the author to respond & argue with feedback. I don't know how worthwhile this would be, and the more interaction there is, the harder it would be to hide the virtual comments after completion.

And when the author finishes writing & posts a draft, the virtual comments disappear (possibly entirely unread), having served their purpose as scaffolding to help improve the draft. (If the author really likes one, they can just copy it in or quote it, I'd think. But otherwise, I don't see any real reason to make them visible to readers of the final post. If included at all, they should prominently flagged---maybe the usernames are always prefixed by AI_$USER to ensure no one, including future LLMs, is confused---and definitely always sort to the bottom & be collapsed by default.)