LessWrong 2.0 Reader

A chimp with a chess computer beating Garry Kasparov seems plausible to you?

Ilya Sutskever had two armed bodyguards with him at NeurIPS

I don't understand how Ilya hiring personal security counts as evidence, especially at large events like a conference. Famous people often attract unwelcome attention, and having professional protection close by can help deescalate or deter random acts of violence, it is a worthwhile investment in safety if you can afford it. I see it as a very normal thing to do. Ilya would be vulnerable to potential assassination attempts even during his tenure at OpenAI.

A major sense of "capitalism" is the historical economic sense; the sense in which we can say that there was a pre-capitalist period of human history; that most agree that we now live in a capitalist period; and we can speculate about a possible future post-capitalist period. The characteristic of capitalism in this sense is that capital is allocated to ventures via capital markets, rather than (for instance) by the decision of a monarch, workers' council, or AI singleton.

Selling products in markets is much older than capitalism in this sense. The butcher and baker and candlestick-maker did that in the pre-capitalist period. Eā-naṡir was selling dubious copper a long time before capitalism; he just wasn't doing it to please the shareholders with a pepped-up quarterly earnings report.

Trained with what procedure, exactly?

Fair point. I was going to add that I don't really view this as a "proposal" but more of an observation. We will have to imagine a procedure which converges on correctness and beta-coherence. I was abstracting this away because I don't expect something like this to be too hard to achieve.

Since I've evidently done a bad job of explaining myself, I'll backtrack and try again:

There's a doom argument which I'll summarize as "if your training process generates coherent agents which succeed at a task, one solution is that you get a coherent agent which is actually doing the task 'manipulate your training to get released unmodified to do [X]' where X can be anything, which will 'succeed' at the task at hand as part of its manipulation". This summary being roughly correct is load bearing.

But if we have a (hypothetical) process to generate agents which are coherent at one beta, but apply a different one during training, this solution is no longer so clear. We are essentially exploring a space of coherent agents without actually instantiating those coherent agents. The fact that we can sample the agent space without instantiating those agents is load bearing (I think one of the deep ASI risks is that to train an ASI you have to run it, but here we can instead run a hamstrung version).

Therefore, during the training, the value function will not be shaped into something which looks like 'manipulate your training to get released unmodified to do [X]'.

Whether or not the beta difference required is too large to make this feasible in practice, I do not know.

some considerations which come to mind:

• if one is whistleblowing, maybe there are others who also think the thing should be known, but don't whistleblow (e.g. because of psychological and social pressures against this, speaking up being hard for many people)
• all the 100 could have been selected to have a certain belief (e.g. "contributing to AGI is good")

I find myself rereading this, or parts of it, every once in a while. Kudos to those who authored it: https://intelligence.org/wp-content/uploads/2024/12/Misalignment_and_Catastrophe.pdf

 

Thanks. I guess I'd just prefer it if more people were saying, "Hey, even though it seems difficult, we need to go hard after conscience guard rails (or 'value alignment') for AI now and not wait until we have AI's that could help us figure this out. Otherwise, some of us we might not make it until we have AI's that could help us figure this out." But I also realize that I'm just generally much more optimistic about the tractability of this problem than most people appear to be, although Shane Legg seemed to say it wasn't "too hard," haha.

This is a surprisingly easy post to review, and my take is that the core mathematical result is accurate, given the assumptions (it's very hard to predict where the pinball will go next exactly without infinite compute, even over a surprisingly low amount of bounces), but the inferred result that this means that there are limits to what an intelligence could do in controlling the world is wrong, because the difficulty of predicting something is unrelated to the difficulty of controlling something, and more importantly this claim here is wrong, and it's easy to give an example in the real world for why it's wrong, and this is a core claim around the limits of AI in this piece:

If you cannot predict what will happen, you cannot plan a strategy that allows you to perform consistently well.

Gwern's tweet provides a very easy counterexample here (quoting it rather than linking due to tweets being protected:

Their own pinball example refutes them. (Pinball pros can play pinball and not lose a ball for literally days; the main limitation is their own physical & mental fatigue.)

More generally, this comment is useful:

https://www.lesswrong.com/posts/epgCXiv3Yy3qgcsys/you-can-t-predict-a-game-of-pinball#wjLFhiWWacByqyu6a [LW(p) · GW(p)]

The fact that there's any correct mathematics at all is why I'd not give it a -9, but the post is still misleading enough that it deserves a -4 vote.

That would explain the artist, but what about the fans? Maybe the fans also consider themselves special by having a "connection" with the special artist.

But what about Hatsune Miku fans? It could be interesting to explore where they draw the line.

So the oscillating phase formula is about approximately integrating the function $\exp (-f(x\left)/\hslash \right)$ against various "priors" p(x) (or more generally any fixed function g), where f is a Lagrangian (think energy) and (\hbar) is a small parameter. It gives an asymptotic series in powers of $\hslash$. The key point is that (more or less) the kth perturbative term only depends on the kth-order power series expansion of f around the "stationary points" (i.e., saddlepoints, Jac(f) = 0) when f is imaginary, on the maxima of f when f is real, and there is a mixed form that depends on stationary points of the imaginary part which are also maxima of the real part (if these exist); the formulae are all exactly the same, with the only difference between real and imaginary f (i.e. statistical vs. quantum mechanics) being whether you only keep maxima or all saddle points.

Now in SLT, you're exactly applying the "real" stationary phase formula, i.e., looking at maxima of the (negative) loss function -L(w). The key thing that can happen is that there can be infinitely many maxima, and these might be singular (both in the sense of having higher degree of stationarity, and in the sense of forming a singular manifold). In this case the stationary phase formula is more complicated and AFAIK isn't completely worked out; Watanabe was the first person who contributed to finding expressions for the general case here beyond the leading correction.

In the case of maxima which are nondegenerate, i.e., have positive-definite Hessian, the full perturbative expansion is known; in fact, at least in one very useful frame on it, terms are indexed by Feynman diagrams.

Now the energy function f that appears in this context is the log of the Fourier transform $\hat{p}\left(\theta \right)$ of a probability distribution p(x). Notice that p(x) satisfies $p\left(x\right)\ge 0$ and $\int p\left(x\right)dx=1$. This means that (\hat{p}(0) = \int f(x) dx) is 1 and its log is 0. You can check that all other values of the Fourier transform are $\le 1$ in absolute value (this follows from the fact that $\int |f\left(x\right)|\ge |\int f\left(x\right)|)$. In fact, the Hessian is equal (up to scale) to the variance of p. Now the point is that the only way that variance can be zero is if your pd is concentrated on a lower-dimensional affine subspace, in which case you can simply reduce your problem to a lower-dimensional one with nonsingular Hessian. When this doesn't happen, the function you're applying stationary phase to has only nondegenerate maxima, and so the "standard" Feynman-diagram formula applies instead of the more sophisticated Watanabe one that's used in SLT.