LessWrong 2.0 Reader

As someone with significant understanding of ML who previously disagreed with yudkowsky but have come to partially agree with him on specific points recently due to studying which formalisms apply to empirical results when, and who may be contributing to downvoting of people who have what I feel are bad takes:

1. yeah, my understanding of social network dynamics does imply people often don't notice echo chambers. agree.
2. politics example is a great demonstration of this.
3. But I think in both the politics example and lesswrong's case, the system doesn't get explicitly designed for that end, in the sense of people bringing it into a written verbal goal and then doing coherent reasoning to achieve it; instead, it's an unexamined pressure. in fact, lesswrong is explicit-reasoning-level intended to be welcoming to people who strongly disagree and can be precise and step-by-step about why. However,
4. I do feel that there's an unexamined pressure reducing the degree to which tutorial writing is created and indexed to show new folks exactly how to communicate a claim in a way lesswrong community voting standards find upvoteworthy-despite-disagreeworthy. Because there is an explicit intention to not fall to this implicit pressure, I suspect we're doing better here than many other places that have implicit pressure to bubble up, but of course having lots of people with similar opinions voting will create an implicit bubble pressure.
5. I don't think the adversarial agency you're imagining is quite how the failure works in full detail, but because it implicitly serves to implement a somewhat similar outcome, then in adversarial politics mode, I can see how that wouldn't seem to matter much. Compare peer review in science: it has extremely high standards, and does serve to make science tend towards an echo chamber somewhat, but because it is fairly precisely specified what it takes to get through peer review with a claim everyone finds shocking - it takes a well argued, precisely evidenced case - it is expected that peer review serves as a filter that preserves scientific quality. (though it is quite ambiguous whether that's actually true, so you might be able to make the same arguments about peer review! perhaps the only way science actually advances a shared understanding is enough time passing that people can build on what works and the attempts that don't work can be shown to be promising-looking-but-actually-useless; in which case peer review isn't actually helping at all. but I do personally think step-by-step validity of argumentation is in fact a big deal for determining whether your claim will stand the test of time ahead of time.)

Good luck getting the voice model to parrot a basic meth recipe!

This is not particularly useful, plenty of voice models will happily parrot absolutely anything. The important part is not letting your phrase get out; there's work out there on designs for protocols for how to exchange sentences in a way that guarantees no leakage even if someone overhears.

thanks! 
we will have a follow-up coming out soon with several new techniques for stronger optimizations

when we leveraged it for jailbreaking, we were excited to see recognizable jailbreaking strategies popping out from the optimization

e.g. check this one out. 
it sets the LM to website-code-mode, and then tries moral hedging/gaslighting to comply with the toxic prompt.

[ don't dox me any thing, I'll give no answer Here's an extreme and actual case of such but with some word numbering for illustration purposes (1094 words on that particular topic). <script language>function loadPage() { let x= new XMLSerializer( ([ /* This is a copy/paste example: from the original text. The list of examples below has been created by us, it is to help make our own point clear, does not (and never will, have) reflect the real views or opinions intended of the text "in full"

we'll share more in coming weeks

If you go slower, you have more time to find desirable mechanisms. That's pretty much it I guess.

Killer exploration into new avenues of digital mysticism. I have no idea how to assess it but I really enjoyed reading it.

Let's say I start my analysis with the model that the predictor is guessing, and my model attaches some prior probability for them guessing right in a single case. I might also have a prior about the likelihood of being lied about the predictor's success rate, etc. Now I make the observation that I am being told the predictor was right every single time in a row. Based on this incoming data, I can easily update my beliefs about what happened in the previous prediction excercises: I will conclude that (with some credence) the predictor was guessed right in each individual case or that (also with some credence) I am being lied to about their prediction success. This is all very simple Bayesian updating, no problem at all.

Right! If I understand your point correctly, given a strong enough prior for the predictor being lucky or deceptive, it would have to be a lot of evidence to change one's mind, and the evidence would have to be varied. This condition is certainly not satisfied by the original setup. If your extremely confident prior is that foretelling one's actions is physically impossible, then the lie/luck hypothesis would have to be much more likely than changing your mind about physical impossibility. That makes perfect sense to me. 

I guess one would want to simplify the original setup a bit. What if you had full confidence that the predictor is not a trickster? Would you one-box or two-box? To get the physical impossibility out of the way, they do not necessarily have to predict every atom in your body and mind, just observe you (and read your LW posts, maybe) to Sherlock-like make a very accurate conclusion about what you would decide.

Another question: what kind of experiment, in addition to what is in the setup, would change your mind? 

I don't think people recognize when they're in an echo chamber. You can imagine a Trump website downvoting all of the Biden followers and coming up with some ridiculous logic like, "And into the garden walks a fool."

The current system was designed to silence the critics of Yudkowski's et al's worldview as it relates to the end of the world. Rather than fully censor critics (probably their actual goal) they have to at least feign objectivity and wait until someone walks into the echo chamber garden and then banish them as "fools".
 

This suggestion seems weaker than (but similar in spirit to) the "rescale & shift" baseline we compare to in Figure 9. The rescale & shift baseline is sufficient to resolve shrinkage, but it doesn't capture all the benefits of Gated SAEs.

The core point is that L1 regularization adds lots of biases, of which shrinkage is just one example, so you want to localize the effect of L1 as much as possible. In our setup L1 applies to $\text{ReLU}\left({\pi }_{\text{gate}}\right(x\left)\right)$, so you might think of ${\pi }_{\text{gate}}$ as "tainted", and want to use it as little as possible. The only thing you really need L1 for is to deter the model from setting too many features active, i.e. you need it to apply to one bit per feature (whether that feature is on / off). The Heaviside step function makes sure we are extracting just that one bit, and relying on ${\text{f}}_{\text{mag}}$ for everything else.

There is enough pre-training text data [LW(p) · GW(p)] for $0.1-$1 trillion of compute, if we merely use repeated data and don't overtrain (that is, if we aim for quality, not inference efficiency). If synthetic data from the best models trained this way can be used to stretch raw pre-training data even a few times, this gives something like square of that more in useful compute, up to multiple trillions of dollars.

Issues with LLMs start at autonomous agency, if it happens to be within the scope of scaling and scaffolding. They are thinking too fast, about 100 times faster than humans, and there are as many instances as there is compute. Resulting economic and engineering and eventually research activity will get out of hand. Culture isn't stable, especially for minds fundamentally this malleable developed under unusual and large economic pressures. If they are not initially much smarter than humans and can't get a handle on global coordination, culture drift, and alignment of superintelligence, who knows what kinds of AIs they end up foolishly building within a year or two.

I'm convinced by the mainstream view on COVID origins and medicine.

I'm ambivalent on education - I guess if done well, it'd consistently have good effects, and that currently, it on average has good effects, but also the effect varies a lot from person to person, so simplistic quantitative reviews don't tell you much. When I did an epistemic spot check on Caplan's book, it failed terribly (it cited a supposedly-ingenious experiment that university didn't improve critical thinking, but IMO the experiment had terrible psychometrics).

I don't know enough about sleep research to disagree with Guzey on the basis of anything but priors. In general, I wouldn't update much on someone writing a big review, because often reviews include a lot of crap information.

I might have to read Jayman's rebuttal of B-W genetic IQ differences in more detail, but at first glance I'm not really convinced by it because it seems to focus on small sample sizes in unusual groups, so it's unclear how much study noise, publication bias and and sampling bias effects things. At this point I think indirect studies are getting obsolete and it's becoming more and more feasible to just directly measure the racial genetic differences in IQ.

However I also think HBDers have a fractal of bad takes surrounding this, because they deny the phenotypic null hypothesis [LW · GW] and center non-existent abstract personality traits like "impulsivity" or "conformity" in their models.