LessWrong 2.0 Reader

Yeah, I admit a lot of the crux comes down to whether thinking whether your case is more the exception or the rule, and I admit that I think that your situation is more unusual compared to the case where you can locally verify something without having to execute the global plan.

I tend to agree far more with Paul Christiano than with John Wentworth on the delta of

But to address what it would mean for alignment to generalize more than capabilities, this would essentially mean it's easier to get an AI to value what you value without the failure modes of deceptive/pseudo/suboptimality alignment than it is to get an AI that actually executes on your values through capabilities in the real world.

(On the highest level I do not know my values and wouldn't hand over the full control of the future to any AI because I don't trust that I could tell good from bad, I think I'd mostly be confused about what it did.)

I admit that I both know a lot more about what exactly I value, and I also trust AIs to generalize more from values data than you do, for several reasons.

I also don't think it's useful to try and learn much about pronouns qua pronouns social battles over them. Using the pronoun people ask you to use has become a proxy for all sorts of other tolerant/benevolent attitudes towards that person and the way they want to live their life, and to an even greater extent, refusing to do that is a proxy for thinking they should be ignored, or possibly reviled, or possibly killed. 

I don't think everyone proxies it that way -- I know there are some people who are just old-fashioned, or passionate about prescriptive grammar, or have essentialist beliefs about gender but are libertarian about others' behavior. I think that if everyone had very high confidence that someone not using the pronouns they requested meant that at worst that person mildly disapproves of them but would still actively defend their civil + legal + human rights, there would probably be a lot less of the handwringing you mention, and we'd be able to learn a lot more about the fundamental intrinsic meaning of pronouns. 

to me "alignment tax" usually only refers to alignment methods that don't cost-effectively increase capabilities, so if 90% of alignment methods did cost effectively increase capabilities but 10% did not, i would still say there was an "alignment tax", just ignore the negatives.

Also, it's important to consider cost-effective capabilities rather than raw capabilities - if a lab knows of a way to increase capabilities more cost-effectively than alignment, using that money for alignment is a positive alignment tax

As I keep saying, deception is not some unique failure mode. Humans are constantly engaging in various forms of deception. It is all over the training data, and any reasonable set of next token predictions. There is no clear line between deception and not deception. And yes, to the extent that humans reward ‘deceptive’ responses, as they no doubt often will inadvertently do, the model will ‘learn’ deception.

https://www.lesswrong.com/posts/a392MCzsGXAZP5KaS/deceptive-ai-deceptively-aligned-ai [LW · GW] 

(found in the comments of this prediction market)

In the limit (what might be considered the ‘best imaginable case’), we might imagine researchers discovering an alignment technique that (A) was guaranteed to eliminate x-risk and (B) improve capabilities so clearly that they become competitively necessary for anyone attempting to build AGI. 

I feel like throughout this post, you are ignoring that agents, "in the limit", are (likely) provably taxed by having to be aligned to goals other than their own.  An agent with utility function "A" is definitely going to be less capable at achieving "A" if it is also aligned to utility function "B".   I respect that current LLM's not best described as having a singular consistent goal function, however, "in the limit" that is what they will be best described as.  

LW Feature request/idea for a feature - In posts that have lots of in-text links to other posts,  perhaps add an LLM 1-2 sentence (context informed) summary in the hover preview? 

I assume that for someone who has been around the forum for many years, various posts are familiar enough that name-dropping them in a link is sufficient to give context. But If I have to click a link and read 4+ other posts as I am going through one post, perhaps the LW UI can fairly easily build in that feature. 

(suggesting it as a features since it does seem like LW is a place that experiments with various features not too different from this - ofc, I can always ask for a LLM summary manually myself if I need to) 

Anyone who harbors such an intense attachment to specific gendered pronoun preferences clearly sees it as much more than a superficial aesthetic designator.

This makes you sound like a bit of a straw vulcan imo. All I have to do is imagine how jarring and upsetting it would be to have everyone start calling me "she" and it's very obvious how, for almost all people, what pronoun others call them is deeply emotionally salient.

GSAI paper club is tomorrow (gcal ticket), summary (by me) and discussion of this paper

You're welcome!

Cheers!

I think you might have implicitly assumed that my main crux here is whether or not take-off will be fast. I actually feel this is less decision-relevant for me than the other cruxes I listed, such as time-to-AGI or "sharp left turns." If take-off is fast, AI alignment/control does seem much harder and I'm honestly not sure what research is most effective; maybe attempts at reflectively stable or provable single-shot alignment seem crucial, or maybe we should just do the same stuff faster? I'm curious: what current AI safety research do you consider most impactful in fast take-off worlds?

To me, agent foundations research seems most useful in worlds where:

• There is an AGI winter and we have time to do highly reliable agent design; or
• We build alignment MVPs, institute a moratorium on superintelligence, and task the AIs to solve superintelligence alignment (quickly), possibly building off existent agent foundations work. In this world, existing agent foundations work helps human overseers ground and evaluate AI output.