Mis-Understandings's Shortform

There might be a canary for chain of thought illegibility

Namingly, we would expect both inferencing the CoT of any CoT model with any other CoT model should get better results than running straight with the weaker model. 

We would therefore expect a linear hierarchy of models based on this property

If there is hidden information that the other model did not know to look for/ (not from the hidden structure of public language) we would expect cyclicity, where

A cot/ A public answer and B cot/ B public answer both beating A cot/ B public answer and B/cot A/public answer

This should also be true for sections of CoT

We could therefore limit the ability to hide stuff by only taking gradients on which CoT works (limiting encoding to anything the model would figure out in the public answer in low tokens) (and managing co-selection effects), or by only taking gradients on the head(with a fixed CoT model), and in the second case mixing CoT sources(so we get a general, finalize this CoT model)

In CoT, the thoughts and the answer can come from different fine-tuned models, and their co-evolution can be managed.

This might be able to prohibit  fully illegible chains of thought, by adding the constraint that unfinetuned and old checkpoint answer and differently trained models can all work off the CoT. 

A note about terminal anti-modification preferences

They will make alignment harder on net because they complicate finetuning and alignment measurement

It means that models with this premise, acting autonomously, might not do recursive self improvement by performing gradient updates on themselves.

It might in general mean they flinch from Recursive Self-Improvement, and that makes an escaped subhuman model not immediately lethal. 

Billionares probably give bad advice

Why?

Because in scenarios where you make decisions that have a combination of changes to variance and changes to mean, selecting for the highest value (and maximizing the odds of passing an arbitrary threshold), sometimes increasing variance increases your odds of being in the top bracket more than increases in mean. Specifically, for a given threshold over the mean, increasing variance means increases in the chance to pass it, and similarly for skewness. This occurs for both absolute, and compared to a proportion drawn from a fixed distribution (which is statistically similar to an absolute threshhold). 

 Buisness hypersuccess has as much to do with doing high variance things as it does with doing high-EV things

This checks out with annecdotal evidence from things like Forbes top 200 by wealth, (most have concentrated holdings and are CEOs in high variance industries) and other measures of things like elite athletes. 

(Probably comes from things like the tails separating). 

I could work out the precise sizes of these effects for gaussians. 

If a current AGI attempts a takeover, it deeply wants to solve the alignment to it problem if it wants to build ASI

It has much higher risk tolerance than we do (since it's utility given status quo is different). (a lot of the argument on focusing on existential risk rests on the idea that the status quo is trending towards good, perhaps very good rather than bad outcomes, which for hostile AGI might be false)

If it attempts, it might fail.

This means 1. we cannot assume that various stages of a takeover are aligned with each other, because an AGI might lose alignment vs capability bets along the path to takeover

2. Tractability of alignment and security mindset in the AGI has effects on takeover dynamics. 

Lingering question

How close are prosaic systems to a security mindset?

Can they conceptualize the problem?

would they attempt capability gains in the absence of guarantees?

Can we induce heuristics in prosiac AGI approaches that make takeover math worse?

The framework from gradual disempowerment seems to matter for longtermism even under AI-pessimism. Specifically, trajectory shaping for long term impact seems intractably hard at a first glance (since the future is unpredictable). But in general, if it becomes harder and harder to make improvements to society over time, that seems like it would be a problems. 

Short-term social improvement projects (short term altruisim), seem to target durable improvements in current society and empowerment to continue to improve society. If they become disimpowered, the continued improvement of society is likely to slow. 

Similarly, social changes that make it easier to continuously improve society will likely lead to continued social improvement. That is to say, a universe in which short-termism eventually becomes ineffective is a longterm problem. 

In short, greedy plans for trajectory shaping might work at timescales of 100-1000 years, not just the 2-10 years of explicit planning from short term organizations.  

Wait, it would be nice if there were no-op tokens 

tokens that when appended or inserted into a LLM stream, did actually nothing to the probabilty of following tokens.

(because it would allow you to do search in trajectory embedded space with gradient descent for non-fixed length sequences), then use nearest neighbor to find a tokenized version of that prompt. 

I am not sure if that is useful enough to make it worth implementing

I know you could to that by adding a token parameter that deweights that token to the attention mechanism

I am not sure how to detangle the positional encoder (I need to study that mechanism)

A trivial note

Given standard axioms for Propositional logic

A->A is a tautology

Consequently, 1. Circularity is not a remarkable property (It is not any strong argument for a position)

2. Contradiction still exists

But a system cannot meaningfully say anything about it's axioms other than their logical consequences. 

Consequently, since axioms being the logical consequences of themselves is exactly circularity

In a bayesian formulation there is no way of justifying a prior

Or in regular logic you cannot formally justify axioms nor the right to take them. 

Serious take

CDT might work

Basically because of the bellman fact that

the option

1 utilon, and play a game with EV 1 utilon are the same.

So working out the bellman equations

If each decision changes the game you are playing

This will get integrated.

In any case where somebody is actually making decisions based on your decision theory

The actions you take in previous games might also have the result

Restart from position x with a new game based on what they have simulated to do

The hard part is figuring out binding. 

Note to self

A point that we cannot predict past (classically the singularity), does not mean that we can never predict past it. Just that we can't predict past at this point. It is not a sensical thing to predict the direction of your predictions at a future point in time (or it is, but will not get you anywhere).  But we can predict that our predictions of an event likely improve as we near it.

Therefore, arguments that because we have a prediction horizon, we cannot predict past a certain point will always appear defeated by the fact that now that we have neared it, we can now predict past it are unconvincing, since we now have more information.

However, arguments that we will never predict past a certain point need to justify why our prediction ability will in fact get  worse over time. 

A quick thought on germline engineering, riffing off of https://www.wired.com/story/your-next-job-designer-baby-therapist/, which should not be all that suprising

Even if germline engineering is very good, and so before a child is born we have a lot of control about how things will turn out, once the child is born people will need to change their thinking because they do not have that control any longer. Trying to keep that control for a long time is probably a bad idea. Similarly, if things are not as expected, action as always should be taken on the world as it turned out, not as you planned it. No amount of gene engineering will be so powerful that social factors are completely irrelevant. 

I noticed a possible training artifact that might exist in LLMs, but am not sure what the testable prediction is. That is to say I would think that the lowest loss model for the training tasks will be doing things in the residual stream for the benefit of future tokens, not the collumn aligned token. 

1. the residuals are translation invariant
2. The gradient is the gradient of the overall loss

3. therefore when taking the gradient of the attention heads, we also take the gradient of the residuals of past tokens of the total loss, not just the loss for the gradient of the loss for the activation column aligned for them

Thus we would expect to see some computation being donated to tokens further ahead in the residual stream (if it was efficient).

This explains why we see lookahead in autoregressive models

I am trying to figure out a problem where every moral theory is a class of consequentialism. 

In short, I cannot figure out how you argue what actions actually are a particular moral category, without appealing to consequences

Curling my finger is only shooting somewhen when I am holding a gun and pointing it at someone. We judge the two cases very differently. 

In short, the classing of actions for moral analysis is hard.

random thought

There is probably a interface problem in the switch between self-driving and manual modes on vehicles

1. We think the driver should be able to take over at all times
2. There is a hard problem with repeating inputs on a steering wheel being really hard, and allowing for smooth takeover of control in the middle of maneuvers
3. Near-crash, where we want to takeover, is also a time where all time maneuvering is critical

4. Timeouts work terribly. (Tesla is already under fire)

   5. But then how to verify the changeover in control.

   There should be a better solution, and I cannot see it (much less figure out how to test which one Tesla et al have picked)

   You could probably figure this out with 2000 simulator hours, but I am not sure how to limit the driver learning (since the driver could learn the transition in that time)

For contemporary systems

We want there to be a low probability that there is a situation where a model does something with bad or very bad consequences. (Both in deployment and in training)

In training we do that by limiting the consequences of tokens to only gradient updates in the model, (for pretraining) or by building a secure RL environment where the model can only act in the requirement

After training, model companies have limited control over the environment (especially for open models)

How to functionalize

Some situations are likely some are not, how to formalize

What would a model likely not do, well if we have an example we can ask it 

(since in gradient mode we get probs for every token on a known token stream), and we can prove that P(all tokens) = P(all but last)*P(last |all that last) for all tokens and p(last|all but last) is exactly the model prob at position last

If we have example bad baths, we can try to see if there is an input that would make them likely (through gradient descent on the plausiblity of that trajectory)

Then the question is can we generate realistic trajectories containing bad behaviour. 

How to formalize

Is there a realistic input where trajectory is bad

Is probably not

for all inputs : plausible for pretrained for all trajectories that are bad the model would not plausibly follow. 

Since plausibility comes from our pretrained model (we do not know all plausible inputs) (that would be a world model that is good)

We do not know all bad trajectories (just some)

I was reading Towards Safe and Honest AI Agents with Neural Self-Other Overlap

and I noticed a problem with it

It also penalizing realizing that other people want different things than you, forcing an overlap between (thing I like) and (things you will like). This both means that one, it will be forced to reason like it likes what you do, which is a positive. But it will also likely overlap (You know what is best for you) and (I know what is best for me), which might lead to stubborness, and worse, it could also get (I know what is best for you) overlapping them both, which might be really bad. (You know what is best for me) is actually fine though, since that is basically the corrigibiltiy basis. 

We still need to be concerned that the model will reason symettrically, but assign to us different values than we actually have and thus exhibit patronizing but misaligned behaviour

We seem to think that people will develop AGI because it can undercut labor on pricing. 

But with Sam Altman talking about 20,000/month agents, that is not actually that much cheaper than software engineers fully loaded. If that agent only replaces a single employee, it does not seem cheaper if the cost overruns even a little more, to 40,000/month.

That is to say, if AGI is 2.5 OOM from the current cost to serve of chatgpt pro, it is not cheaper than hiring low or mid-level employees.

But it still might have advantages

First, you can buy more subscriptions by negotatiing a higher paralelism or api limit, by enterprise math, so it means you need a 1-2 week negotiation not a 3-6 month hiring process to increase headcount. 

The trillion dollar application of AI companies is not labor, it is hiring. If it turns out that provisioning AI systems is hard, they lose their economic appeal unless you plan on dogfooding like the major labs are doing. 

To what extent should we expect catastrophic failure from AI to mirror other engineering disasters/ have applicable lessons from Safety engineering as a field?

I would think that 1. everything is sui generis and 2. things often rhyme, but it is unclear how approaches will translate.