LessWrong 2.0 Reader

I welcome being held accountable for this going wrong in various ways.

It happening at all already constitutes “going wrong”.

Also: by what means can you be “held accountable”?

If this is true, then it’s a damning indictment of Less Wrong and the authors who post here, and is an excellent reason not to read anything written here.

Tax (and other frictions) is pretty well-known as a market distortion.  There is some symmetry to the taxation - both sides of the market have the same friction (tax if win, no tax if loss), so this will make it less noticeable.  And in robust markets, there will be enough professional involvement (who get taxed on net winnings, not on wins ignoring losses), that the effect is probably small.

Still, thanks for pointing it out - the effects are real, and important.

But then, who does carry that responsibility? No-one.

For thie case of this particular feature and ones like it: The LessWrong team. And, in this case, more specifically, me.

I welcome being held accountable for this going wrong in various ways. (I plan to engage more with people who present specific cruxes rather than a generalized "it seems scary", but, this seems very important for a human to be in the loop about, who actually takes responsibility for both locally being good, and longterm consequences)

Reasoning is:

• Currently it takes 40-60 seconds to generate jargon (we've experimented with ways of trimming that down but it's gonna be at least 20 seconds)
• I want authors to actually review the content before it goes live.
• Once authors publish the post, I expect very few of them to go back and edit it more.
• If it happens automagically during draft saving, then by the time you get to "publish post", there's a natural step where you look at the autogenerated jargon, check if it seems reasonable, approve the ones you like and then hit "publish"
• Anything that adds friction to this process I expect to dramatically reduce how often authors bother to engage with it.

[The Simpsons] "I'm Not Made of Stone !", Says Krusty the Clown

Sorry, risk aversion.

Also, the usual situation of "if I think the main proposition is unlikely, bad outcomes will be dominated by cases where I miss loopholes in the bet or otherwise lose the bet for reasons unrelated to the truth of the proposition".

(edit: thank you for your comment! I genuinely appreciate it.)

"""I think (not sure!) the damage from people/orgs/states going "wow, AI is powerful, I will try to build some" is larger than the upside of people/orgs/states going "wow, AI is powerful, I should be scared of it"."""
^^ Why wouldn't people seeing a cool cyborg tool just lead to more cyborg tools? As opposed to the black boxes that big tech has been building?
I agree that in general, cyborg tools increase hype about the black boxes and will accelerate timelines. But it still reduces discourse lag. And part of what's bad about accelerating timelines is that you don't have time to talk to people and build institutions --- and, reducing discourse lag would help with that.

"""By not informing the public that AI is indeed powerful, awareness of that fact is disproportionately allocated to people who will choose to think hard about it on their own, and thus that knowledge is more likely to be in reasonabler hands (for example they'd also be more likely to think "hmm maybe I shouldn't build unaligned powerful AI")."""
^^ You make 3 assumptions that  I disagree with:
1) Only reasonable people who think hard about AI safety will understand the power of cyborgs
2) You imply a cyborg tool is a "powerful unaligned AI", it's not, it's a tool to improve bandwidth and throughput between any existing AI (which remains untouched by cyborg research) and the human
3) That people won't eventually find out. One obvious way is that a weak superintelligence will just build it for them. (I should've made this explicit, that I believe that capabilities overhang is temporary, that inevitably "the dam will burst", that then humanity will face a level of power they're unaware of and didn't get a chance to coordinate against. (And again, why assume it would be in the hands of the good guys?))

Thanks Akash for this substantive reply!

Last minute we ended up cutting a section at the end of the document called "how does this [referring to civics, communications, coordination] all add up to preventing extinction?" It was an attempt to address the thing you're pointing at here:

I think there's something tricky about a document that has the vibe "this is the most important issue in the world and pretty much everyone else is approaching it the wrong way" and then pivots to "and the right way to approach it is to post on Twitter and talk to your friends."

Sadly we didn't feel we could get the point across well enough to make our timing cutoff for v1. A quick attempt at the same answer here (where higher context might make it easier to convey the point):

• One way someone could be asking "does this all add up" is "are we going to survive?" And to that, our answer is mostly "hmm, that's not really a question we think about much. Whether we're going to make it or not is a question of fact, not opinion. We're just trying our best to work on what we think is optimal."
• The other way someone could be asking "does this all add up" is "is this really a good plan?" That's a great question -- now we're talking strategy.
• There are of course huge things that need to be done. A number of the authors support what's written in A Narrow Path, which offers very ambitious projects for AI policy. This is one good way to do strategy: start with the "full plan" and then use that as your map. If your plan can't keep us safe from superintelligence even in the ideal case where everything is implemented, then you need a new plan. (This is one of many of our RSP concerns -- what is the full plan? Most everything after "we detect the dangerous thing" still seems to require the level of intervention described in A Narrow Path)
• Communication, coordination, and civics straightforwardly don't add up to A Narrow Path's suggestions. However, they are bottlenecks. (Why we think this is communicated somewhat in the doc, but there's a lot more to say on this). This is another good way to do strategy: look at things that are required in any good plan, and optimize for those. We don't see a winning world where AGI risks are not a global common knowledge, with people aware and concerned and acting at a scale far larger scale than today.
• Where A Narrow Path does backprop from what's needed to actually stop superintelligence from being built, this doc presents more of "what can we do immediately, today." We try to tie the "What we can do immediately" to those bottlenecks that we think are needed in any plan. 

And yes - It was written more with a low-context AIS person in mind, and most of the high-context people we try to redirect towards "hey, reach out to us if you'd like to be more deeply involved." I think v2 should include more suggestions for bigger projects, that people with more context can pursue. Would love to hear your (or others') views on good projects. 

Also, great comment about the people who have done the most on communication so far. I really commend their efforts and writing something about this is definitely something v2 can include. 

On joining government organizations... I'm just speaking for myself on this one as I think my coauthors have different views on governance than myself. Yes-- this is good and necessary. Two caveats: 

• Be willing to go against the grain. It seems the default path right now is for governments to support the same "reactive framework" that AGI companies are pushing. I'm worried about this, and I think we need people in government positions and advising them that are much more frank about the risks and unwilling to go for "convenient" solutions that fit the overton window. If the necessary safety regulations don't fit the current overton window, then the overton window has to change, not the regulation. Huge props to CAIS for SB1047 and whatever future bill efforts follow from them or others.
• Be willing to help. Lots of people in government do care, and simply don't know what's going on. Try to be helpful to them before assuming they're antagonistic to x-risk. I've met lots of government people who are very amenable to a "hey, I'd be super happy to talk you through the technical details of AI, and explain why some people are worried about x-risk." Non-threatening, non-asking-for-something approaches really work. 

More to say later - to this and other comments in the thread. For now, taking the weekend to get a bit of rest :) 

This sounds like the shortest, most succinct description of what I consider 1 central problem in the entire AI alignment community that makes a lot of their other thoughts on AI risk very distorted, so thank you for stating the issue so clearly.

I don't think this is the only problem, and there is another problem I'll mention in this comment, but as a description of why I tend to be more optimistic about data-based alignment strategies than many others, this statement is why I think this way:

It also part of the genre of arguments that argue that AI systems should act in particular ways regardless of their domain, training data, and training procedure -- which I think by now we should have extremely strong priors against, given that for literally all AI systems -- and I mean literally all, including MCTS-based self-play systems -- the data from which the NN's learn is enormously important for what those NNs learn. 

My post about the issue of assuming instrumental convergence being far more unconstrained than happens in reality is also another central issue in the AI alignment community:

https://www.lesswrong.com/posts/HHkYEyFaigRpczhHy/ai-88-thanks-for-the-memos#EZLm32pKskw8F5xiF [LW(p) · GW(p)]