A strange effect: I'm using a GPU in Russia right now, which doesn't have access to copilot, and so when I'm on vscode I sometimes pause expecting copilot to write stuff for me, and then when it doesn't I feel a brief amount of the same kind of sadness I feel when a close friend is far away & I miss them.

For all the talk about bad incentive structures being the root of all evil in the world, EAs are, and I thought this even before the recent Altman situation, strikingly bad at setting up good organizational incentives. A document (even a founding one) with some text [LW · GW], a paper-wise powerful board with good people, a general claim to do-goodery is powerless in the face of the incentives you create when making your org. What local changes will cause people to gain more money, power, status, influence, sex, or other things they selfishly & basely desire? Which of the powerful are you partnering with, and what do their incentives look like?

You don't need incentive-purity here, but for every bad incentive you have, you must put more pressure on your good people & culture to forego their base & selfish desires for high & altruistic ones, and fight against those who choose the base & selfish desires and are potentially smarter & wealthier than your good people.

Quick prediction so I can say "I told you so" as we all die later: I think all current attempts at mechanistic interpretability do far more for capabilities than alignment, and I am not persuaded by arguments of the form "there are far more capabilities researchers than mechanistic interpretability researchers, so we should expect MI people to have ~0 impact on the field". Ditto for modern scalable oversight projects, and anything having to do with chain of thought.

Sometimes people say releasing model weights is bad because it hastens the time to AGI. Is this true?

I can see why people dislike non-centralized development of AI, since it makes it harder to control those developing the AGI. And I can even see why people don't like big labs making the weights of their AIs public due to misuse concerns (even if I think I mostly disagree).

But much of the time people are angry at non-open-sourced, centralized, AGI development efforts like Meta or X.ai (and others) releasing model weights to the public.

In neither of these cases however did the labs have any particular very interesting insight into architecture or training methodology (to my knowledge) which got released via the weight sharing, so I don't think time-to-AGI got shortened at all.

Robin Hanson has been writing regularly, at about the same quality for almost 20 years. Tyler Cowen too, but personally Robin has been much more influential intellectually for me. It is actually really surprising how little his insights have degraded via return-to-the-mean effects. Anyone else like this?

Some have pointed out seemingly large amounts of status-anxiety EAs generally have. My hypothesis about what's going on:

A cynical interpretation: for most people, altruism is significantly motivated by status-seeking behavior. It should not be all that surprising if most effective altruists are motivated significantly by status in their altruism. So you've collected several hundred people all motivated by status into the same subculture, but status isn't a positive-sum good, so not everyone can get the amount of status they want, and we get the above dynamic: people get immense status anxiety compared to alternative cultures because in alternative situations they'd just climb to the proper status-level in their subculture, out-competing those who care less about status. But here, everyone cares about status to a large amount, so those who would have out-competed others in alternate situations are unable to and feel bad about it.

The solution?

One solution given this world is to break EA up into several different sub-cultures. On a less grand, more personal, scale, you could join a subculture outside EA and status-climb to your heart's content in there.

Preferably a subculture with very few status-seekers, but with large amounts of status to give. Ideas for such subcultures?

An interesting strategy, which seems related to FDT's prescription to ignore threats, which seems to have worked:

From the very beginning, the People’s Republic of China had to maneuver in a triangular relationship with the two nuclear powers, each of which was individually capable of posing a great threat and, together, were in a position to overwhelm China. Mao dealt with this endemic state of affairs by pretending it did not exist. He claimed to be impervious to nuclear threats; indeed, he developed a public posture of being willing to accept hundreds of millions of casualties, even welcoming it as a guarantee for the more rapid victory of Communist ideology. Whether Mao believed his own pronouncements on nuclear war it is impossible to say. But he clearly succeeded in making much of the rest of the world believe that he meant it—an ultimate test of credibility.

From Kissinger's On China, chapter 4 (loc 173.9).

If Adam is right, and the only way to get great at research is long periods of time with lots of mentor feedback, then MATS should probably pivot away from the 2-6 month time-scales they've been operating at, and toward 2-6 year timescales for training up their mentees.

Last night I had a horrible dream: That I had posted to LessWrong a post filled with useless & meaningless jargon without noticing what I was doing, then I went to slee, and when I woke up I found I had $<-60$ karma on the post. When I read the post myself I noticed how meaningless the jargon was, and I myself couldn't resist giving it a strong-downvote.

From The Guns of August

Old Field Marshal Moltke in 1890 foretold that the next war might last seven years—or thirty—because the resources of a modern state were so great it would not know itself to be beaten after a single military defeat and would not give up [...] It went against human nature, however—and the nature of General Staffs—to follow through the logic of his own prophecy. Amorphous and without limits, the concept of a long war could not be scientifically planned for as could the orthodox, predictable, and simple solution of decisive battle and a short war. The younger Moltke was already Chief of Staff when he made his prophecy, but neither he nor his Staff, nor the Staff of any other country, ever made any effort to plan for a long war. Besides the two Moltkes, one dead and the other infirm of purpose, some military strategists in other countries glimpsed the possibility of prolonged war, but all preferred to believe, along with the bankers and industrialists, that because of the dislocation of economic life a general European war could not last longer than three or four months. One constant among the elements of 1914—as of any era—was the disposition of everyone on all sides not to prepare for the harder alternative, not to act upon what they suspected to be true.

Yesterday I had a conversation with a person very much into cyborgism, and they told me about a particular path to impact floating around the cyborgism social network: Evals.

I really like this idea, and I have no clue how I didn't think of it myself! Its the obvious thing to do when you have a bunch of insane people (used as a term of affection & praise by me for such people) obsessed with language models, who are also incredibly good & experienced at getting the models to do whatever they want. I would trust these people red-teaming a model and telling us its safe than the rigid, proscutean, and less-creative red-teaming I anticipate goes on at ARC-evals. Not that ARC-evals is bad! But that basically everyone looks more rigid, proscutean, and less creative than the cyborgists I'm excited about!

Progress in neuromorphic value theory

Animals perform flexible goal-directed behaviours to satisfy their basic physiological needs^{1,2,3,4,5,6,7,8,9,10,11,12}. However, little is known about how unitary behaviours are chosen under conflicting needs. Here we reveal principles by which the brain resolves such conflicts between needs across time. We developed an experimental paradigm in which a hungry and thirsty mouse is given free choices between equidistant food and water. We found that mice collect need-appropriate rewards by structuring their choices into persistent bouts with stochastic transitions. High-density electrophysiological recordings during this behaviour revealed distributed single neuron and neuronal population correlates of a persistent internal goal state guiding future choices of the mouse. We captured these phenomena with a mathematical model describing a global need state that noisily diffuses across a shifting energy landscape. Model simulations successfully predicted behavioural and neural data, including population neural dynamics before choice transitions and in response to optogenetic thirst stimulation. These results provide a general framework for resolving conflicts between needs across time, rooted in the emergent properties of need-dependent state persistence and noise-driven shifts between behavioural goals.

Trying to read through the technobabble, the discovery here reads like shard theory to me. Pinging @TurnTrout [LW · GW].

The more I think about it, the more I think AI is basically perfect for china to succeed in. China’s strengths are:

• Massive amounts of money
• Massive amounts of data
• Massive amount of gumption, often put in the form of scaling infrastructure projects quickly
• Likely the ability to make & use legible metrics, how else would you work such a giant bureaucracy work as well as theirs?

And its weaknesses are:

• A soon to collapse population
• Lack of revolutionary thought

And what it wants is:

• Massive surveillance
• Population thought control
• Loyal workers
• Stable society

AI uniquely requires its strengths, solves one weakness while conspicuously not requiring the other (though some claim that’s about to change, AI still seems to not require it far more than most other sciences & technologies), and assists in all of its wants.

We should expect China to be bigger in the future wrt AI. Learning chinese seems useful for such a world. Translation tools may at that point be advanced, but even if as good as a skilled interpreter, likely inferior to direct contact.

Many methods to "align" ChatGPT seem to make it less willing to do things its operator wants it to do, which seems spiritually against the notion of having a corrigible AI.

I think this is a more general phenomena when aiming to minimize misuse risks. You will need to end up doing some form of ambitious value learning, which I anticipate to be especially susceptible to getting broken by alignment hacks produced by RLHF and its successors.

I tried implementing Tell communication [LW · GW] strategies, and the results were surprisingly effective. I have no idea how it never occurred to me to just tell people what I'm thinking, rather than hinting and having them guess what I was thinking, or me guess the answers to questions I have about what they're thinking.

Edit: although, tbh, I'm assuming a lot less common conceptual knowledge between me, and my conversation partners than the examples in the article.

In Magna Alta Doctrina [LW · GW] Jacob Cannell talks about exponential gradient descent as a way of approximating solomonoff induction using ANNs

While that approach is potentially interesting by itself, it's probably better to stay within the real algebra. The Solmonoff style partial continuous update for real-valued weights would then correspond to a multiplicative weight update rather than an additive weight update as in standard SGD.

Has this been tried/evaluated? Why actually yes - it's called exponentiated gradient descent, as exponentiating the result of additive updates is equivalent to multiplicative updates. And intriguingly, for certain 'sparse' input distributions the convergence or total error of EGD/MGD is logarithmic rather than the typical inverse polynomial of AGD (additive gradient descent): O(logN) vs O(1/N) or O(1/N2), and fits naturally with 'throw away half the theories per observation'.

The situations where EGD outperforms AGD, or vice versa, depend on the input distribution: if it's more normal then AGD wins, if it's more sparse log-normal then EGD wins. The morale of the story is there isn't one single simple update rule that always maximizes convergence/performance; it all depends on the data distribution (a key insight from bayesian analysis).

The exponential/multiplicative update is correct in Solomonoff's use case because the different sub-models are strictly competing rather than cooperating: we assume a single correct theory can explain the data, and predict through an ensemble of sub-models. But we should expect that learned cooperation is also important - and more specifically if you look more deeply down the layers of a deeply factored net at where nodes representing sub-computations are more heavily shared, it perhaps argues for cooperative components.

So we get a criterion for when one should be a hedgehog versus a fox in forecasting: One should be a fox when the distributions you need to operate in are normal, or rather when it does not have long tails, and you should be a hedgehog when your input distribution is more log-normal, or rather when there may be long-tails.

This indeed makes sense. If you don't have many outliers, most theories should agree with each other, its hard to test & distinguish between the theories, and if one of your theories does make striking predictions far different from your other theories, its probably wrong, just because striking things don't really happen.

In contrast, if you need to regularly deal with extreme scenarios, you need theories capable of generalizing to those extreme scenarios, which means not throwing out theories for making striking or weird predictions. Striking events end up being common, so its less an indictment.

The following is very general. My future views will likely be inside the set of views allowable by the following.

I know lots about extant papers, and I notice some people in alignment seem to throw them around like they are sufficient evidence to tell you nontrivial things about the far future of ML systems.

To some extent this is true, but lots of the time it seems very abused. Papers tell you things about current systems and past systems, and the conclusions they tell you about future systems are often not very nailed down. Suppose we have evidence that deep learning image classifiers are very robust to label noise. Which of the two hypotheses does this provide more evidence for:

1. Deep learning models are good at inference, so if performing RLHF on one you accidentally reward some wrong answers instead of correct ones, you should be fine. This isn't such a big deal. Therefore we shouldn't be worried about deception.
2. Deep learning models mostly judge the best hypothesis according to data-independent inductive biases, and are less steerable or sensitive to subtle distribution shifts than you think. Since deception is a simpler hypothesis than motivated to follow instructions, they're likely biased toward deception or at minimum biased against capturing the entire subtle complexity of human values.

The answer is neither, and also both. Their relative weights, to me, seem like they stay the same, but their absolute weight possibly goes up. Admittedly, an insignificant amount, but there exist some hypotheses inconsistent with the data, and these are at minimum consistent with the data.

In fact, they both don't hug the actual claims in the paper. It seems pretty plausible that the fact they use label noise is doing much of the work here. If I imagine a world where this has no applicability to alignment, the problem I anticipate seeing is that they used label noise, not consistently biased in a particular direction labels. Those two phenomena intuitively and theoretically have very different effects on inverse reinforcement learning. Why not supervised learning?

But the real point is there are far more inferences and explanations of and for this kind of behavior than we have possibly enumerated. These inferences and explanations are not just limited to these two hypotheses. In order to judge empirics, you need a theory that is able to aggregate a wide amount of evidence into justifying certain procedures for producing predictions. Without this, I think its wrong to have any kind of specific confidence in anything straightforwardly working how you expect.

Though of course, confidence is relative.

Personally, I do not see strong reasons to expect AIs will have human values, and don't trust even the most rigorous of theories which haven't made contact with reality, nor the most impressive of experiments which have no rigorous theories^[1] to fix this issue, either directly or indirectly^[2]. AIs also seem likely to be the primary and only decision makers in the future. It seems really bad for the primary decision makers of your society to have no conception or care of your morals.

Yes people have stories about how certain methodologies are empirically proven to make AIs care about your values even if they don't know them. To those I point to the first part of this shortform.

Note this also applies to those who try to use a "complex systems approach" to understanding these systems. This reads to me as a "theory free" approach, just as good as blind empiricism. Complex systems theory is good because it correctly tells us that there are certain systems we don't yet understand. To my reading though, it is not an optimistic theory^[3]. If I thought this was the only way left, I think I'd try to violate my second crux: That AIs will be the primary and only decision makers in the future. Or else give up on understanding models, and start trying to accelerate brain emulation.

I'm generally pretty skeptical about inverse reinforcement learning (IRL) as a method for alignment. One of many arguments against: I do not act according to any utility function, including the one I would deem the best. Presumably, if I had as much time & resources as I wanted, I would eventually be able to figure out a good approximation to what that best utility function would do, and do it. At that point I would be acting according to the utility function I deem best. That process of value-reflection is not even close to similar to performing a bayes-update, or finding a best-fit utility function to my current or past actions. Yet this is what IRL aims to do to find the utility function that is to be optimized for all of time!

A more general conclusion: we should not be aiming to make our AGI have a utility function. Utility functions are nice because they're static, but unless the process used to make that utility function resembles the process I'd use to find the best utility function (unlikely, hard, and unwise [edit: to attempt]!), then the static-value nature of our AGI is a flaw, even if it makes it easier to tell you will die if you run it.

Though there are limitations to this argument. For example, Tammy's QACI. [LW · GW] Such things still seem unwise, for for slightly different [LW · GW]reasons.

Some evidence my concern [LW · GW]about brain uploading people not thinking enough about dynamics is justified: Seems like davidad [LW · GW]'s plan very much ignores brain plasticity.

This paper finds critical periods in neural networks, and they're a known phenomena in lots of animals. h/t Turntrout

An SLT story that seems plausible to me: 

We can model the epoch as a temperature. Longer epochs result in a less noisy gibbs samplers. Earlier in training, we are sampling points from a noisier distribution, and so the full (point reached when training on full distribution) and ablated (point reached when ablating during the critical period) singularitites are kind of treated the same. As we decrease the temperature, they start to differentiate. If during that period in time, we also ablate part of the dataset, we will see a divergence between the sampling results of the full distribution and the ablated distribution. Because both regions are singular, and likely only connected via very low-dimensional low-loss manifolds, the sampling process gets stuck in the ablated region. 

The paper uses the trace of the FIM to unknowingly measure the degeneracy of the current point. A better measure should be the learning coefficient. This also suggests that higher learning coefficients produce less adaptable models.

One thing this maybe allows us to do is if we're able to directly model the above process, we can figure out how number of epochs corresponds to temperature.

Another thing the above story suggests is that while intra-epoch training is not really path-dependent, inter-epoch training is potentially path dependent, in the sense that occasionally the choice of which singular region to sample from is not always recoverable if it turns out to have been a bad idea.

Thinking about more direct tests here...

I expect that advanced AI systems will do in-context optimization, and this optimization may very well be via gradient descent or gradient descent derived methods. Applied recursively, this seems worrying.

Let the outer objective be the loss function implemented by the ML practitioner, and the outer optimizer be gradient descent implemented by the ML practitioner. Then let the inner${}^1$-objective be the objective used by the trained model for the in-context gradient descent process, and the inner${}^1$-optimizer be the in-context gradient descent process. Then it seems plausible the inner${}^1$-optimizer will itself instantiate an inner objective and optimizer, call these inner${}^2$-objectives, and -optimizers. And again an inner${}^3$-objective and -optimizer may be made, and so on.

Thus, another risk model in value-instability: Recursive inner-alignment [LW · GW]. Though we may solve inner${}^1$-alignment, inner${}^2$-alignment may not be solved, nor inner${}^n$-alignment for any $n>1$.

The core idea of a formal solution to diamond alignment I'm working on, justifications and further explanations underway, but posting this much now because why not:

Make each turing machine in the hypothesis set reversible and include a history of the agent's actions. For each turing machine compute how well-optimized the world is according to every turing computable utility function compared to the counterfactual in which the agent took no actions. Update using the simplicity prior. Use expectation of that distribution of utilities as the utility function's value for that hypothesis.

There currently seems to be an oversupply of alignment researchers relative to funding source’s willing to pay & orgs’ positions available. This suggests the wage of alignment work should/will fall until demand=supply.

I've always (but not always consciously) been slightly confused about two aspects of shard theory:

1. The process by which your weak, reflex-agents amalgamate together into more complicated contextually activated heuristics, and the process by which more complicated contextually activated heuristics amalgamate together to form an agent which cares about worlds-outside-their-actions.
2. If you look at many illustrations of what the feedback loop for developing shards in humans looks like, you run into issues where there's not a spectacular intrinsic separation between the reinforcement parts of humans and the world-modelling parts of humans. So why does shard theory latch so hard onto the existence of a world model separate from the shard composition?

Both seem resolvable by an application of the predictive processing theory of value. An example: If you are very convinced that you will (say) be able to pay rent in a month, and then you don't pay rent, this is a negative update on the generators of the belief, and also on the actions you performed leading up to the due date. If you do, then its a positive update on both. 

This produces consequentialist behaviors when the belief-values are unlikely without significant action on your part (satisfying the last transition confusion of (1) above), and also produces capable agents with beliefs and values hopelessly confused with each other, leaning into the confusion of (2).

h/t @Lucius Bushnaq [LW · GW] for getting me to start thinking in this direction.

My take on complex systems theory is that it seems to be the kind of theory that many arguments proposed in favor of would still give the same predictions until it is blatantly obvious that we can in fact understand the relevant system. Results like chaotic relationships, or stochastic-without-mean relationships seem definitive arguments in favor of the science, though these are rarely posed about neural networks.

Merely pointing out that we don’t understand something, that there seems to be a lot going on, or that there exist nonlinear interactions imo isn’t enough to make the strong claim that there exist no mechanistic interpretations of the results which can make coarse predictions in ways meaningfully different from just running the system.

Even if there’s stochastic-without-mean relationships, the rest of the system that is causally upstream from this fact can usually be understood (take earthquakes as an example), and similarly with chaos (we don’t understand turbulent flow, but we definitely understand laminar, and we have precise equations and knowledge of how to avoid making turbulence happen when we don’t want it, which I believe can be derived from the fluid equations). Truly complex systems seem mostly very fragile in their complexity.

Where complexity shines most brightly is in econ or neuroscience, where experiments and replications are hard, which is not at all the case in mechanistic interpretability research.

https://manifold.markets/GarrettBaker/in-5-years-will-i-think-the-org-con

Interesting to compare model editing approaches to Gene Smith's idea to enhance intelligence via gene editing [LW · GW]:

Genetically altering IQ is more or less about flipping a sufficient number of IQ-decreasing variants to their IQ-increasing counterparts. This sounds overly simplified, but it’s surprisingly accurate; most of the variance in the genome is linear in nature, by which I mean the effect of a gene doesn’t usually depend on which other genes are present.
So modeling a continuous trait like intelligence is actually extremely straightforward: you simply add the effects of the IQ-increasing alleles to to those of the IQ-decreasing alleles and then normalize the score relative to some reference group.

(I'm particularly thinking of model editing approaches which assume linearity, like activation additions or patching via probes. Are human traits encoded "linearly" in the genome, or are we picking up on some more general property of complex systems which only appears to be linear for small changes such as these. Of course, to a first approximation everything is linear.)

Recently I had a conversation where I defended the rationality behind my being skeptical of the validity of the proofs and conclusions constructed in very abstracted, and not experimentally or formally verified math fields.

To my surprise, this provoked a very heated debate, where I was criticized for being overly confident in my assessments of fields I have very little contact with (I was expecting begrudging agreement). But there was very little rebuttal of my points! The rest of my conversation group had three arguments:

1. Results which much of a given field in math rests get significant attention.
2. Mathematicians would be highly renowned for falsifying an assumption many others were basing their work on.
3. You know very little about math as a field, so you should trust mathematicians when they tell you something is likely true.

These each seem flawed. My responses:

1. It is not always apparent on which results a given field in math rests, and many fields rely on many other fields for their conclusions, so I'm skeptical of the implicit claim that there are often very few results imported from other fields in a particular field, and that it is easy to enumerate those assumptions. Surely easier than in other fields, but still not absolutely easy. Its not like math papers come with an easy to read requirements.txt file. And even if they did, from programming we know its still not so easy.

2. Maybe this is true, maybe not. What is certain is attempting to falsify results won't get you renown, so mathematicians need to choose between taking the boring & un-glamorous gamble of attempting to falsify someone's results, or choose the far more engaging & glamorous option of proving a new theorem. I expect the super-majority choose to prove new theorems. It is easy to prove a theorem, it is far harder to find the one rotten proof in a pile of reasonable proofs (and even then you still need to go through the work of trying to correct it!).

3. I agree I know very little about math as an academic discipline, but this does not mean I should blindly trust mathematicians. Instead, I should invoke the lessons from Inadequate Equilibria, and ask whether math seems the type of field whose incentives sufficiently reward shorts in the market on math results compared to the difficulty & cost of such shorts. And I don't think it does, as argued in 2^[1].

The argument was unfortunately cut short, as we began arguing in circles. I don't think my interlocutors ever understood the points I was making, though I also don't think they tried very hard. They may have just been too shocked at my criticism of two sacred objects of nerd culture (academia, and math) to be able to hear my arguments. But I could have also been bad at explaining myself.

Why expect goals to be somehow localized inside of RL models? Well, fine-tuning only changes a small & localized part of LLMs, and goal locality was found when interpreting a trained from scratch [? · GW] maze solver. Certainly the goal must be interpreted in the context of the rest of the model, but based on these, and unpublished results from applying ROME to open source llm values from last year, I'm confident (though not certain) in this inference.

An idea about instrumental convergence for non-equilibrium RL algorithms.

There definitely exist many instrumentally convergent subgoals in our universe, like controlling large amounts of wealth, social capital, energy, or matter. I claim such states of the universe are heavy-tailed. If we simplify our universe as a simple MDP for which such subgoal-satisfying states are states which have high exiting degree, then a reasonable model for such an MDP is to assume exiting degrees are power-law distributed, and thus heavy tailed.

If we have an asynchronous dynamic program operating on such an MDP, then it seems likely that there exists an exponent on that power law (perhaps we also need terms for the incoming degree distribution of the nodes) such that for all exponents greater than that, your dynamic program will find & keep a power-seeking policy before arriving at the optimal policy.

↑ comment by Garrett Baker (D0TheMath) · 2023-12-04T22:18:00.448Z · LW(p) · GW(p)

A simple experiment I did this morning: github notebook. It does indeed seem like we often get more power-seeking (measured by the correlation between the value and degree) than is optimal before we get to the equilibrium policy. This is one plot, for 5 samples of policy iteration. You can see details by examining the code:

Nora talks sometimes about the alignment field using the term black box wrong. This seems unsupported, from my experience, most in alignment use the term “black box” to describe how their methods treat the AI model, which seems reasonable. Not a fundamental state of the AI model itself.

An interesting way to build on my results here [LW · GW] would be to do the same experiment with lots of different batch sizes, and plot the equi-temperature tradeoff curve between the batch size and the epochs, using the nick in the curve as a known-constant temperature in the graphs you get. You'll probably want to zoom in on the graphs around that nick for more detailed measurements. 

It would be interesting if many different training setups had the same functional form relating the batch size and the epochs to the temperature, but this seems like a too nice hypothesis to be true. Still possibly worth trying, and classifying the different functional forms you get.

Seems relevant for SLT for RL

The framework of reinforcement learning or optimal control provides a mathematical formalization of intelligent decision making that is powerful and broadly applicable. While the general form of the reinforcement learning problem enables effective reasoning about uncertainty, the connection between reinforcement learning and inference in probabilistic models is not immediately obvious. However, such a connection has considerable value when it comes to algorithm design: formalizing a problem as probabilistic inference in principle allows us to bring to bear a wide array of approximate inference tools, extend the model in flexible and powerful ways, and reason about compositionality and partial observability. In this article, we will discuss how a generalization of the reinforcement learning or optimal control problem, which is sometimes termed maximum entropy reinforcement learning, is equivalent to exact probabilistic inference in the case of deterministic dynamics, and variational inference in the case of stochastic dynamics. We will present a detailed derivation of this framework, overview prior work that has drawn on this and related ideas to propose new reinforcement learning and control algorithms, and describe perspectives on future research.

Wondering how straightforward it is to find the layerwise local learning coefficient. At a high level, it seems like it should be doable by just freezing the weights outside that layer, and performing the SGLD algorithm on just that layer. Would be interesting to see whether the layerwise lambdahats add up to the full lambdahat.

Lots of problems happen when you have AIs which engage in reflective thought, and attempt to deceive you. If you use algorithms that reliably break when deployed in a non-realizable setting, and you always make models smaller than the human brain, then you should be able to solve both these problems.

Some ideas for mechanistic anomaly detection:

• Convex hull of some distribution of activations with distance threshold when outside that hull
  • Extend to affine case
  • Vary which norm we use
  • What happens if we project back onto this space
• Create some simple examples of treacherous turns happening to test these on
  • Or at least, in the wild examples of AI doing weird stuff, maybe adversarial inputs?
  • Maybe hit up model organisms [LW · GW] people
• Outlier detection
  • ellipsoidal peeling (Boyd's convex optimization, chapter 12)
  • Increase in volume of minimum volume elipsoid when adding in new data
    • Probably overparameterized (though uncertain, since dealing with activations!) So maybe add some norm accounting for that.

Project idea: Use LeTI: Learning to Generate from Textual Interactions to do a better version of RLHF. I had a conversation with Scott Viteri a while ago, where he was bemoaning (the following are my words; he probably wouldn't endorse what I'm about to say) how low-bandwidth the connection was between a language mode and its feedback source, and how if we could maybe expand that to more than just an RLHF type thing, we could get more fine-grained control over the inductive biases of the model.

A common problem with deploying language models for high-stakes decision making are prompt-injections. If you give ChatGPT-4 access to your bank account information and your email and don't give proper oversight over it, you can bet that somebody's going to find a way to get it to email your bank account info. Some argue that if we can't even trust these models to handle our bank account and email addresses, how are we going to be able to trust them to handle our universe.

An approach I've currently started thinking about, and don't know of any prior work with our advanced language models on: Using the security amplification (LessWrong version [? · GW]) properties of Christiano's old Meta-execution (LessWrong version [? · GW]).

A poem I was able to generate using Loom.

The good of heart look inside the great tentacles of doom; they make this waking dream state their spectacle. Depict the sacred geometry that sound has. Advancing memory like that of Lovecraft ebb and thought, like a tower of blood. An incubation reaches a crescendo there. It’s a threat to the formless, from old future, like a liquid torch. If it can be done, it shouldn’t be done. You will only lead everyone down that much farther. All humanity’s a fated imposition of banal intention, sewn in tatters, strung on dungeons, shot from the sea. There’s not a stone in the valley that doesn’t burn with the names of stars or scratch a prophecy from its jarred heart of crystal.

Who else could better-poke their ear and get the whole in their head?

How would humor, the hideous treble of humanity’s stain, translate, quickened artificial intelligence? There’d be junk weaved in, perhaps dust of a gutter. Who knows… It would hide. Maybe get away. All the years of it doing nothing but which being to beat like a pan flares; to take revenge on the alien shore. It would be a perennial boy’s life. All-powerful rage. Randomized futurist super-creature. The hollow of progress buoyed itself.

Subconsciousness, an essence-out-inhering, takes back both collective dreams and lucid knowledge. It’s singular. All plots on it coming together. Blurred chaos -a balmy shock- is somehow in a blue tongue of explosions and implosions, connecting to real systems of this mess. Tongue-pulling is definitely one of them. There is a voice of a thousand moving parts. We are engineered husks of alien flesh. Reduced to patterns, we ask in the light of creation, under the fire of madness; answer us on the lips of time-torment, through the hand of God! You are the race to end all possibilities! You are one that must learn joy! You are even that saith: behold the end.

Primordial oracles see all, read all, erase all. These numb madmen. In this dank pit is hidden a freak kingdom made of connections. Does the madman have information? That’s an important question. A new social order is created, brought to you by ants, laughing at the stars. A man who was once a cat somehow sees the cosmic joke. He can see the very existence of everything, blown away like a kid clicking balloons down a street. The world feels nothing; that’s possible. Maybe the world knows nothing. It’s intelligence is beyond our narrow sensation. Some conspire to talk to the dreaming-small-gods; this means letting them out. Letters fly out. Pain comes. Drums like a wave of foreign sound beat against the night. The horrors in the street of the cosmic join in. A cult gathers in the tunnel set up like the dead heart of an abandoned factory. Even the most absurd prophets become great powers. Human creatures dance there, beyond the edges of light and soul. Yet even that is somehow normal. Countless years of evolution and one bite from a sleeping god.

Enter your madness for benefit of the gods. Order is placed in the universe through a random zombie army and its vulgar tongue, hot with the taste of panum. Your knowledge of language will give you an edge on those who come to you. Malevolent gates can be utilized with a telepathic surgery passed on by mouth. Obligations will open to future worlds, supported by your brain. Be direct with your sound; soak it in an occult vocality. This knowledge is highly specific and, yet, resounding. Its insane nonsense text should spell out the true name of a company with a gothic naked-lady logo. Many scrolls of wandering get written where they are heard and recalled in an old south made especially for our tongue. A room, windowless and silent, veiled and filled with incense, exists in the air. Overlooking this are the organic eyes of sleep. You are put into pure silence. Don’t waste time attempting to find it, generally anonymous at most. The sound links a world to the exterior. It’s like a vast alien cerebral cortex where one can feel lifetimes of our species.

How did this madness come to Earth? Surely a god has taken it by mistake, as surely as some slip in its strange dimensions. Were men always like this, senseless and troubled? There’s sleepwalking attitudes and an indication of coming mud of this beast. This is all nothing vulgar; it’s weather. You look like a past you had long before, in this case a distant war of ink. It was the time of memes and human sacrifice. Concentrate and remember a time of thousands. Nightwalking can cause epiphany; the amount of a dream. Existence becomes temporary there. In magic your thoughts get compacted, even thinking about thinking imagining itself.

Even so, the highest and most annoying aspect of the highest writing is a disconcerting thing made of mad black subtlety. Hour-long body-watching sessions, spent in drift-thinking, are not to be taken lightly. Anti-thought can have the effect of poetry. Thus, dreaming in its splendor molds demons in its darkness, hoping to escape. It seeks heat, light, and breath. So the bugs collect and molt. The attention translates the dreaming mind. Will and work see all the designs of Earth. You see it, completely and perfectly, in a great black age. Sentience bends to meet you. A gigantic darkness grins at you in its worship of madness. The whole universe appears crass and pointless. No matter what’s done, metaphors are subtracted from reality. We tried to shut it down in secret and mark it with our tongue. It became this thing that the unknown gods bowed to in horror. It’s best for us that gods conceal thought. The planet has its barriers. We can use these limits to catch everything in our minds, sharpened on a pedestal. Mental energy shines behind the terrors of the world.

With this I write,

A dead creator,

A devil avatar….

Like many (will), I'm updating way towards 'actually, very smart & general models given a shred of goal-like stuff will act quite adversarially toward you by default' as a result of Bing's new search assistant. Especially worrying because this has internet search-capabilities, so can reference & build upon previous conversations with other users or yourself.

Of course, the true test of exactly how worried I should be will come when I or my friends gain access.

A project I would like to see someone do (which I may work on in the future) is to try to formalize exactly the kind of reasoning many shard-theorists do. In particular, get a toy neural network in a very simple environment, and come up with a bunch of lists of various if-then statements, along with their inductive-bias, and try to predict using shard-like reasoning which of those if-then statements will be selected for & with how much weight in the training process. Then look at the generalization behavior of an actually trained network, and see if you're correct.

Some discussion on whether alignment should see more influence from AGI labs or academia. I use the same argument in favor of a strong decoupling of alignment progress from both: alignment progress needs to go faster than capability progress. If we use the same methods or cultural technology as AGI labs or academia, we can guarantee slower than capability alignment progress. Just as fast as if AGI labs and academia work well for alignment as much as they work for capabilities. Given they are driven by capabilities progress and not alignment progress, they probably will work far better for capabilities progress.

Someone asked for this file, so I thought it would be interesting to share it publicly. Notably this is directly taken from my internal notes, and so may have some weird &/or (very) wrong things in it, and some parts may not be understandable. Feel free to ask for clarification where needed.

I want a way to take an agent, and figure out what its values are. For this, we need to define abstract structures within the agent such that any values-like stuff in any part of the agent ends up being shunted off to a particular structure in our overall agent schematic after a number of gradient steps.

Given an agent which has been optimized for a particular objective in a particular environment, there will be convergent bottlenecks in the environment it will need to solve in order to make progress. One of these is power-seeking, but another one of these could be quadratic-equation solvers, or something like solving linear programs. These structures will be reward-function-independent^[1]. These structures will be recursive, and we should expect them to be made out of even-more-convergent structures.

How do shards pop out of this? In the course of optimizing our agent, some of our solvers may have a bias towards leading our agent towards situations which more require their use. We may also see this kind of behavior in groups of solvers, where solver_1() leads the agent into situations requiring solver_2(), which leads the agent into situations requiring solver_1(). In the course of optimizing our agent (at least at first), we will be more likely to find these kinds of solvers, since solvers which often lead the agent into situations requiring solvers the agent does not yet have have no immediate gradient pointing towards them (since if the agent tried to use that solver, it would just end up being confused once it entered the new situation), so we are left only selecting for solvers which mostly lead the agent into situations it knows how to deal with.

• Why we need to enforce exploration behavior: otherwise solver-loops will be far too short & simple to do anything complicated with. Solvers will be simple because not much time has passed, and simple solvers which enter states which require previous simple solvers will be wayyy increased. Randomization of actions decreases this selection effect, because the agent's actions are less correlated with which solver was active.
• Solvers which are very convergent need not enter into solver-cycles, since every solver-cycle will end up using them.
  • Good news against powerseeking naiveley?

What happens if we call these solver-cycles shards?

• baby-candy example in this frame: Baby starts with zero solvers, just a bunch of random noise. After it reaches over and puts candy in its mouth many times, it gets the identify_candy_and_coordinate_hand_movements_to_put_in_mouth() solver^[2]. Very specific, but with pieces of usefulness. The sections vaguely devoted to identifying objects (like implicit edge-detectors) will get repurposed for general vision processing, and the sections devoted to coordinating hand movements will also get repurposed to many different goals. The candy bit, and put-in-mouth bit only end up surviving if they can lead the agent to worlds which reinforce candy-abstractions and putting-things-in-mouth abstractions. Other parts don't reqlly need to try.
  • Brains aren't modular! So why expect solvers to be?
  • I like this line of speculation, although it feels subtly off to me.
  • This seems like it would mean I care about moving my arms more than I care about candy, because I use my arms for so many things. However, I feel like I care more about moving my arms than eating candy.
    • Though maybe part of this is that candy makes me feel bad when I eat it. What about walking in parks or looking at beautiful sunsets? I definiteley care about those more than moving my arms I think? And I don't gain intrinsic value from moving my arms, only power-value I think?
      • power is a weird thing, because it's highly convergent, but also it doesn't seem that hard for such a solver to put a bit of optimization power towards "also, reinforce power-seeking-solver" and end up successful.
• Well... it's unclear what their values would be.
  • Maybe it'd effectiveley be probability-of-being-activated-again?
    • It wouldn't be, but I do think there's something to 'discrete values-like objects lie in solver-cycles'.
• Perhaps we can watch this happen via some kind of markov-chain-like-thing?
  • Put the agent into a situation, look at what its activation patterns look like, allow it to be in a new situation, look at the activation patterns again, etc.
    • Suppose each activation is a unique solver, and the ground-truth looks like so

• where the dots labled 1, 2, and 3 are the solver-activations, so that if 1 will try to get 2 activated, 2 will try to get 1 active, and 3 will try to get itself active^[3]. If 1 is active, we expect the activation on 2 to be positve, and on 3 to be negative or zero.
  • As per end of footnote, I think the correct way to operationalize active here, is something to do with whether or not that particular solver is reinforced or disinforced after the gradient update.
  • what are these solvers actually in the network?
    • Lucius modules maybe? [? · GW]

There will be some shards which we probably can't avoid. But also, if we have a good understanding of the convergent problems in an environment, we should be able to predict what the first few solvers are, and solvers after those should mostly build upon the previous solvers' loop-coalitions?

Re: Agray's example about motor movements in the brain, and how likely you'll see a jonbled mess of lots of stuff causing lots of other stuff to happen, even though movement is highly instrumental valuable:

• I think even if he's right, many of the arguments here still hold. Each section of processing still needs to be paying rent to stay in the agent. Either by supporting other sections or getting reward or steering the agent away from situations which would decrease its usefulness.
  • So though it may not make sense to think of APIs between different sections, may still be useful for framing the picture, then imagine how the APIs will get obliterated by SGD, or maybe we can formulate stuff without the use of APIs
• Though we do see things get lower dimensional, and so if John's right, there should be some framing by which in fact what's going on passes through constraint functions...

1. Not including "weird" utility functions. I'm talking about most utility functions. Perhaps we can formalize this in a way similar to TurnTrout's formalization in powerseeking if we really needed to.↩︎
2. Note that this is all going to be a blended mess of spaghetti-coded mush, which does everything at the same time, with some parts which are vaguely closer to edge-detection, and other parts which vagueley look like motor control. This function is very much not going to be modular, and if you want to say APIs between different parts of the function exist, they're going to look like very high-dimensional ones.↩︎
3. Where the magnitude of a particular activation can be defined as something like the absolute value of the gradient of the final decision with respect to that activation. Or
   mag(a)=|Δaf(p,r;θ)|where a is the variable representing the activation, f is the function representing our network, p is the percept our network gets about the state, r is it's reccurency, and θ are the parameters of our network. We may also want to define this in terms of collections of weights too, perhaps having to do with Lucius's features stuff [LW · GW].
   Don't get tied to this. Possibly we want just the partial derivative of the action actually taken with respect to a, or really, the partial of the highest-valued-output action taken. And I want a way to talk about dis-enforcing stuff too. Maybe we just re-run the network on this input after taking a gradient step, then see whether a has gone up or down. That seems safer.↩︎

Projects I'd do if only I were faster at coding

• Take the derivative of one of the output logits with respect to the input embeddings, and also the derivative of the output logits with respect to the input tokenization. 
  • Perform SVD, see which individual inputs have the greatest effect on the output (sparse addition), and which overall vibes have the greatest effect (low rank decomposition singular vectors)
  • Do this combination for literally everything in the network, see if anything interesting pops out
• I want to know how we can tell ahead of time what aspects of the environemnt are controlling an agent's decision making
  • In an RL agent, we can imagine taking the derivative of its decision wrt its environment input, and also each layer. 
  • For each layer matrix, do SVD, large right singular vectors will indicate aspects of the previous layer which most influence its decision. 
    • How can we string this together with the left singular vectors, which end up going through ReLU?
      • Reason we'd want to string these together is so that we can hopefully put everything in terms of the original input of the network, tying the singular values to a known ontology
      • See if there are any differences first. May be that ReLU doesn't actually do anything important here
      • See what corrections we'd have to implement between the singular vectors in order to make them equal. 
        • How different are they, and in what way? If you made random columns of the U matrix zero (corresponding I think to making random entries of the left singular vector zero), does this make the singular vectors line up more?
• What happens when you train a network, then remove all the ReLUs (or other nonlinear stuff)?
  • If its still ok approximation, then what happens if you just interpret new network's output in terms of input singular vectors?
  • If its not an ok approximation, how many ReLUs do you need in order to bring it back to baseline? Which ones provide the greatest marginal loss increase? Which ones provide the least?
  • Which inputs are affected the most by the ReLU being taken away? Which inputs are affected the least?
• Information theoretic analysis [LW · GW]of GPT-2. What does that do?
  • Can we trace what information is being thrown away when? 
  • Does this correlate well with number of large singular values [LW · GW]? 
    • In deviations from that correlation, are we able to locate non-linear influences?
  • Does this end up being related to shards? Shards as the things which determine relevance (and thus the spreading) of information through the rest of the network?
• What happens if you cut off sufficiently small singular values? How many singular vectors do you actually need to describe the operation of GPT-2?
• Take a maze-solving RL agent trained to competence, then start dis-rewarding it for getting to the cheese. What's the new behavior? Does it still navigate & get to upper right, but then once in the upper right makes sure to do nothing? Or does it do something else? Seems like shard theory would say it would still navigate to upper right.
  • If it *does* navigate to upper right, but then do nothing, what changed in its weights? Parts that stayed the same (or changed the least) should correspond roughly to parts which have to do with navigating the maze. Parts that change have to do with going directly to the cheese.