LessWrong 2.0 Reader

An example implementation of this feature is Gwern.net's "link-bibliographies" (eg). We extract all URLs from the Markdown, filter, turn them into a list with the available metadata like title/author/tags/abstract, and because we assign IDs to all links, we can also provide a reverse/backlink '↑' to the context in the page where the link was used (and a link might be used multiple times). Wikipedia links are included, but stuffed into a sublist at the end because they would drown out the regular links. It uses dynamic/lazy transclusion, so it doesn't cost anything if you never look at it, but if you want to print out the page or something, that should also be doable as they get loaded then by the print-mode.

We also plan to include a second version of the link-bibliography, a 'browsing history' version, which quietly logs each link you interact with in a big list at the end of the page (we'll probably put it before the full link-bibliography). So the standard full link-bibliography provides all the URLs, but the browsing-history would provide just the shortlist of URLs you interact with, in the order you interacted with them. The idea is that you could more freely move in and out of popups if you didn't have the anxiety of 'losing' them, because there's an append-only log, and after reading a page, you might skim the browsing-history and open up some of them for further reading or to jog your memory about what you were reading at one point. Since the links are in temporal order, it should be easy to reconstruct your train of thought at any point as you were reading. (You could also use it to create a sort of 'custom bibliography', where you pop up a small subset of links focused on some particular claim or thesis, and you can save that to PDF or something.) Since it's all transcludes, the browsing-history is also effectively free (you already paid the cost of downloading & rendering each entry when you popped it up the first time).

Regarding the GPU export restrictions, the U.S. government is trying to outcompete China. However, it seems that Big Tech CEOs, including Sam Altman, have something else in mind. They figure that as long as the U.S. can develop ASI within the next five years—before China recovers from the export restrictions—that’s all that matters. And once it’s built, they don’t really care what happens next: even if X/S-risks arise, it’s still better if it’s their own country that causes them rather than China. If the whole world is going to be dragged into it, they’d rather do it themselves.

There is no war in the run-up to AGI that would derail the project, e.g. by necessitating that most resources be used for capabilities instead of safety research.

Assuming short timelines, I think it’s likely impossible to reach my desired levels of safety culture.

I feel obliged to note that a nuclear war, by dint of EMPs wiping out the power grid, would likely remove private AI companies as a thing for a while, thus deleting their current culture. It would also lengthen timelines.

Certainly not ideal in its own right, though.

Done, thanks!

Sorry; I'm not in the habit of reading the notifications, so I didn't see the "@" tag.

I don't have a good answer (which doesn't change the underlying bad prospects for securing the data). I think I'd tend to prefer to "mitigating risks after potential model theft", because I believe "convince key actors" is fundamentally futile. The kind of security you'd need, if it's possible, would basically shut them down. Which is equivalent to abandoning the "key actor" role to whoever does not implement that kind of security.

Unfortunately, "key actors" would also have to be convinced to "mitigate risks", which they're unlikely to do because that would require them to accept that their preventative measures are probably going to fail. So even the relatively mild "go ahead and do it, but don't expect it to work" is probably not going to happen.

Well, OK, but you also said "actually helps humanity", which assumes some kind of outside view. And you used "aligned" without specifying any particular one of the conflicting visions of "alignment" that are out there.

I absolutely agree that "aligned with whom" is a huge issue. It's one of the things that really bugs me about the word.

I do also agree that there are going to be irreconcilliable differences, and that, barring mind surgery to change their opinions, many people will be unhappy with whatever happens. That applies no matter what an AI does, and in fact no matter what anybody who's "in charge" does. It applies even if nobody is in charge. But if somebody is in charge, it's guaranteed that a lot of people will be very angry at that somebody. Sometimes all you can change is who is unhappy.

For example, a whole lot of Christians, Muslims, and possibly others believe that everybody who doesn't wholeheartedly accept their religion is not only wrong, but also going to suffer in hell for eternity. Those religions are mutually contradictory at their cores. And a probably smaller but still large number of athiests believe that all religion is mindrot that intrinsically reduces the human dignity of anybody who accepts it.

You can't solve that, no matter how smart you are. Favor one view and the other view loses. Favor none, and the other views say that a bunch of people are seriously harmed, even if it's voluntary. It doesn't even matter how you favor a view. Gentle persuasion is still a problem. OK, technically you can avoid people being mad about it after the fact by extreme mind surgery, but you can't reconcile their original values. You can prevent violent conflict by sheer force, but you can't remove the underlying issue.

Still, a lot of the approaches you describe are are pretty ham-handed even if you agree with the underlying values. Some of the desired outcomes you list even sound to me like good ideas... but you ought to be able to work toward those goals, even achieve them, without doing it in a way that pisses off the maximum possible number of people. So I guess I'm reacting to the extreme framing and the extreme measures. I don't think the Taliban actively want people to be mad.

[Edited unusually heavily after posting because apparently I can't produce coherent, low-typo text in the morning]

Basically agree, but not an useful comment.

I'd nuance that as that being alive and energetic is fun -- but when my body no longer grants energy, it's like death already. Say I'm trying to take notes about the content of this thread, but I'm so tired I barely produce anything. If the terms of my body are such that I must first do a timeskip to tomorrow to get more energy, then I want the timeskip.

I guess I understand becoming sleep-deprived and staying up anyway if you don't notice your IQ dropping...

Oh, it is probably my mistake XD I'm also not native. I meant increase, not that it is the maximum it could be, sorry.

I really appreciate this post, as much as it's making me feel that I and everyone I care about have terminal cancer with only 12-60 months to live.

I found the idea that a pivotal act is necessary as especially valuable and expanded on it in my post [Is AI Alignment Enough?](https://www.lesswrong.com/posts/tdrK7r4QA3ifbt2Ty/is-ai-alignment-enough)

Thanks for asking! I said in a later shortform [LW(p) · GW(p)] that I was trying to do too many things in this post, with only vague relationships between them, and I'm planning to split it into pieces in the future.

Your 1-3 are mostly correct. I'd comment as follows:

1. (and also kind of 3) That advice of using the tempered local Bayesian posterior (I like the term -- let's shorten it to TLBP) is mostly aimed at non-SLT researchers (but may apply also to some SLT experiments). The suggestion is not to compute expectations. Rather, just running a single experiment at a weight sampled from the TLBP. The result is analogous to tuning a precision dial on your NN to noise away all circuits for which the quotient (usefulness)/(description length) is bounded above by 1/t (where usefulness is measured in reduction of loss). At t = 0, you're adding no noise and at $t=\infty$ you're fully noising it. 

   This is interesting to do in interp experiments for two general reasons: 

   1. You can see whether the behavior your experiment finds is general or spurious. The higher the temperature range it persists over, the more general it is in the sense of usefulness/description length (and all else being equal, the more important your result is).
   2. If you are hoping to say that a behavior you found, e.g. a circuit, is "natural from the circuit's point of view" (i.e., plausibly occurs in some kind of optimal weight- or activation-level description of your model), you need to make sure your experiment isn't just putting together bits of other circuits in an ad-hoc way and calling it a circuit. One way to see this, that works 0% of the time, is to notice that turning this circuit on or off affects the output on exactly the context/ structure you care about, and has absolutely no effect at all on performance elsewhere. This never works because our interp isn't at a level where we can perform uber-precise targeted interventions, and whenever we do something to a network in an experiment, this always significantly affects loss on unrelated inputs. By having a tunable precision parameter (as given by the TLBP for example), you have more freedom to find such "clean" effects that only do what you want and don't affect loss otherwise. In general, in an imprecise sense, you expect each "true" circuit to have some "temperature of entanglement" with the rest of the model, and if this circuit is important enough to survive tempering to this temperature of entanglement, you expect to see much cleaner and nicer results in the resulting tempered model.
2. In the above context, you rarely want to use the Watanabe temperature or any other temperature that only depends on the number of samples n, since it's much too low in most cases. Instead, you're either looking for a characteristic temperature associated with an experiment or circuit (which in general will not depend on n much), or fishing for behaviors that you hope are "significantly general". Here the characteristic temperature associated with the level of generality that "is not literally memorizing" is the Watanabe temperature or very similar, but it is probably more interesting to consider larger scales.
3. (maybe more related to your question 1): Above, I explained why I think performing experiments at TLBP weight values is useful for "general interp". I also explain that you sometimes have a natural "characteristic temperature" for the TLBP that is independent of sample number (e.g. meaningful at infinite samples), which is the difference between the loss of the network you're studying and a SOTA NN, which you think of as that "true optimal loss". In large-sample (highly underparameterized) cases, this is probably a better characteristic temperature than the Watanabe temperature, including for notions of effective parameter count: indeed, insofar as your NN is "an imperfect approximation of an optimal NN", the noise inherent in this imperfection is on this scale (and not the Watanabe scale). Of course there are issues with this PoV as less expressive NN's are rarely well-conceptualized as TLBP samples (insofar as they find a subset of a "perfect NN's circuits", they find the easily learnable ones rather than the maximally general ones). However it's still reasonable to think of this as a first stab at the inherent noise scale associated to an underparametrized model, and to think of the effective parameter count at this scale (i.e., free energy / log temperature) as a better approximatin of some "inherent" parameter count.