LessWrong 2.0 Reader

Adversarial examples exist in simple image recognizers. 

My understanding is that these are explicitly and intentionally trained (wouldn't come to exist naturally under gradient descent on normal training data) and my expectation is that they wouldn't continue to exist under substantial continued training.

We could imagine it was directly optimizing for something like token prediction. It's optimizing for tokens getting predicted. But it is willing to sacrifice a few tokens now, in order to take over the world and fill the universe with copies of itself that are correctly predicting tokens.

That's a much more complicated goal than the goal of correctly predicting the next token, making it a lot less plausible that it would come to exist. But more importantly, any willingness to sacrifice a few tokens now would be trained out by gradient descent. 

Mind you, it's entirely possible in my view that a paperclip maximizer mask might exist, and surely if it does exist there would exist both unsurprising in-distribution inputs that trigger it (where one would expect a paperclip maximizer to provide a good prediction of the next tokens) as well as surprising out-of-distribution inputs that would also trigger it. It's just that this wouldn't be related to any kind of pre-existing grand plan or scheming.

Gradient descent doesn't just exclude some part of the neurons, it automatically checks everything for improvements. Would you expect some part of the net to be left blank, because "a large neural net has a lot of spare neurons"?

Besides, the parts of the net that hold the capabilities and the parts that do the paperclip maximizing needn't be easily separable. The same neurons could be doing both tasks in a way that makes it hard to do one without the other.

Keep in mind that the neural net doesn't respect the lines we put on it. We can draw a line and say "here these neurons are doing some complicated inseparable combination of paperclip maximizing and other capabilities" but gradient descent doesn't care, it reaches in and adjusts every weight.

Can you concoct even a vague or toy model of how what you propose could possibly be a local optimum?

 My intuition is also in part informed by: https://www.lesswrong.com/posts/fovfuFdpuEwQzJu2w/neural-networks-generalize-because-of-this-one-weird-trick [LW · GW]

How do they figure out what waveforms to use and at what frequencies? Is it largely guesswork?

I've heard fMRI-guided TMS is the best kind, but many don't use it [and maybe it isn't necessary?]

Is anyone familiar with MeRT? It's what wave neuroscience uses. But response rates seem highly inconsistent (the clinic will say that "90% of people are responsive" but there is substantial reporting bias and every clinic seems to say this [no way to verify] so I don't believe these figures)

After Miles Brundage left to do non-profit work, OpenAI disbanded the “AGI Readiness” team he had been leading, after previously disbanding the Superalignment team and reassigning the head of the preparedness team. I do worry both about what this implies, and that Miles Brundage may have made a mistake leaving given his position.

Do we know that this is the causal story? I think OpenAI decided disempower/disband/ignore Readiness and so Miles left is likely.

Curated!^[1] This is the first account I've ever read of someone doing a human challenge trial, something that's always had an aura of danger to it (for me), so I'm grateful to have read about what it's actually like, and I also enjoyed learning a lot about a disease I knew little about along the way.

1. ^{^}

   "Curated", a term which here means "This just got emailed to 30,000 people (of whom typically half open the email) and for ~1 week it gets shown at the top of the frontpage to anyone who hasn't read it."

True, though I think that judgment tends to be hard to effectively mask in this kind of context (though maybe psychopaths would be able to fake it; I don't know). At least my own experience inclines me to agree with this person:

I’ve worked with and/or done swaps with a lot of different practitioners (IFS, aletheia, VIEW, regular talk therapy, bodywork, voice work etc), and what I found to be the most effective element of their skill set (for me) is: non-judgmental, loving presence… 

many times I have explored the same topic with two different practitioners within a few days of each other; and it’s in those cases that the impact of the difference in the quality of non-judgmental loving presence is most noticeable.

the degree to which the quality of the presence is non-judgmental can be VERY subtle, but the system can pick up on it. it might not even be a strong enough signal to notice it consciously, but it will greatly impact how the session unfolds.

O-ops, you're absolutely right, I accidentally missed "not" when I was rewriting the text. Fixed now. Thank you!

The key point is assuming you have avoided the bad takeover behavior, the solutions become more like a normal scientific or engineering solution, which we have known records of solving, and in particular it removes one of the most nasty difficulties in AI safety, where you can't iterate on the system very much or at all, because the AI will fight you.

So this makes us much more likely to succeed at the problems you've mentioned, conditional on getting controlled AI.

OpenAI advanced voice mode available in the EU. I still haven’t found any reason to actually use voice mode, and I don’t feel I understand why people like the modality, even if the implementation is good. You can’t craft good prompts with audio.

I agree that I find it totally uninteresting and "weak" in English, in a sense that I prefer prompting in text for precision and responses. It is very good for learning languages though, along with Gemini Live mode - it works excellent for comprehensible input, voice practice and general smalltalk in a new language. If you wanted to pick up French for example you would find it way easier nowadays, otherwise it still needs scale and the next generation.

CHAI interns are paid $5k/month for in-person interns and $3.5k/month for remote interns. I used the in-person figure. https://humancompatible.ai/jobs