Posts
Comments
Yep.
Specifically, it's named for the papers HiPPO: Recurrent Memory with Optimal Polynomial Projections and How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections, which kicked off the whole transformers for state-space modelling thing
"HiPPO" abbreviates higher-order polynomial projection operators.
tpyo
However, if it is the case that the difference between humans and monkeys is mostly due to a one-shot discrete difference (ie language), then this cannot necessarily be repeated to get a similar gain in intelligence a second time.
Perhaps language is a zero-one, i.e. language renders a mind "cognitively complete" in the sense that the mind can represent anything about the external world, and make any inferences using those representations. But intelligence is not thereby zero-one because intelligence depends on continuous variables like computional speed, memory, etc.
More concretely, I am sceptic that "we end up with AI geniuses, but not AI gods", because running a genius at 10,000x speed, parallelised over 10,000x cores, with instantaneous access to the internet does (I think) make an AI god. A difference is quantity is a difference in kind.
Thar said, there might exist plausible threat models which require an AI which doesn't spatiotemporally decompose into less smart AIs. Could you sketch one out?
chatbots don't map scenarios to actions, they map queries to replies.
Thanks for the summary.
- Does machievelli work for chatbots like LIMA?
- If not, which do you think is the sota? Anthropic's?
Sorry for any confusion. Meta only tested LIMA on their 30 safety prompts, not the other LLMs.
Figure 1 does not show the results from the 30 safety prompts, but instead the results of human evaluations on the 300 test prompts.
- Yep, I agree that MMLU and Swag aren't alignment benchmarks. I was using them as examples of "Want to test your models ability at X? Then use the standard X benchmark!" I'll clarify in the text.
- They tested toxicity (among other things) with their "safety prompts", but we do have standard benchmarks for toxicity.
- They could have turned their safety prompts into a new benchmark if they had ran the same test on the other LLMs! This would've taken, idk, 2–5 hrs of labour?
- The best MMLU-like benchmark test for alignment-proper is https://github.com/anthropics/evals which is used in Anthropic's Discovering Language Model Behaviors with Model-Written Evaluations. See here for a visualisation. Unfortunately, this benchmark was published by an Anthropic which makes it unlikely that competitors will use it (esp. MetaAI).
Clarifications:
The way the authors phrase the Superficial Alignment Hypothesis is a bit vague, but they do offer a more concrete corollary:
If this hypothesis is correct, and alignment is largely about learning style, then a corollary of the Superficial Alignment Hypothesis is that one could sufficiently tune a pretrained language model with a rather small set of examples.
Regardless of what exactly the authors mean by the Hypothesis, it would be falsified if the Corollary was false. And I'm arguing that the Corollary is false.
(1-6) The LIMA results are evidence against the Corollary, because the LIMA results (post-filtering) are so unusually bare (e.g. no benchmark tests), and the evidence that they have released is not promising.
(7*) Here's a theoretical argument against the Corollary:
- Base models are harmful/dishonest/unhelpful because the model assigns significant "likelihood" to harmful/dishonest/unhelpful actors (because such actors contributed to the internet corpus).
- Finetuning won't help because the small set of examples will be consistent with harmful/dishonest/unhelpful actors who are deceptive up until some trigger.
- This argument doesn't generalise to RLHF and ConstitutionalAI, because these break the predictorness of the model.
Concessions:
The authors don't clarify what "sufficiently" means in the Corollary, so perhaps they have much lower standards, e.g. it's sufficient if the model responds safely 80% of the time.
Nope, no mention of xrisk — which is fine because "alignment" means "the system does what the user/developer wanted", which is more general than xrisk mitigation.
But the paper's results suggest that finetuning is much worse than RLHF or ConstitutionalAI at this more general sense of "alignment", despite the claims in their conclusion.
In a controlled human study, responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43% of cases;
I'm not sure how well this metric tracks what people care about — performance on particular downstream tasks (e.g. passing a law exam, writing bugless code, automating alignment research, etc)
"Directly or indirectly" is a bit vague. Maybe make a market on Manifold if one doesn't exist already.
Thanks! I've included Erik Hoel's and lc's essays.
Your article doesn't actually call for AI slowdown/pause/restraint, as far as I can tell, and explicitly guards off that interpretation —
This analysis does not show that restraint for AGI is currently desirable; that it would be easy; that it would be a wise strategy (given its consequences); or that it is an optimal or competitive approach relative to other available AI governance strategies.
But if you've written anything which explicitly endorses AI restraint then I'll include that in the list.
well-spotted😳
Thanks, Zach!
yep my bad
Finite context window.
Countries that were on the frontier of the Industrial Revolution underwent massive economic, social, and political shocks, and it would've been better if the change had been smoothed over about double the duration.
Countries that industrialised later also underwent severe shocks, but at least they could copy the solutions to those shocks along with the technology.
Novel general-purpose technology introduces problems, and there is a maximum rate at which problems can be fixed by the internal homeostasis of society. That maximum rate is, I claim, at least 5–10 years for ChatGPT-3.5.
ChatGPT-3.5 would've led to maybe a 10% reallocation of labour — this figure doesn't just include directly automated jobs, but also all the second- and third-order effects. ChatGPT-4, marginal on ChatGPT-3.5 would've led to maybe a 4% reallocation of labour.
It's better to "flatten the curve" of labour reallocation over 5–10 years rather than 3 months because massive economic shocks (e.g. unemployment) have socio-economic risks and costs.
Sure, the "general equilibrium" also includes the actions of the government and the voting intentions of the population. If change is slow enough (i.e. below 0.2 OOMs/year) then the economy will adapt.
Perhaps wealth redistribution would be beneficial — in that case, the electorate would vote for political parties promising wealth redistribution. Perhaps wealth redistribution would be unbeneficial — in that case, the electorate would vote for political parties promising no wealth redistribution.
This works because electorial democracy is a (non-perfect) error-correction mechanism. But it operates over timescales of 5–10 years. So we must keep economic shocks to below that rate.
Great idea! Let's measure algorithmic improvement in the same way economists measure inflation, with a basket-of-benchmarkets.
This basket can itself be adjusted over time so it continuously reflected the current use-cases of SOTA AI.
I haven't thought about it much, but my guess is the best thing to do is to limit training compute directly but adjust the limit using the basket-of-benchmarks.
The economy is a complex adaptative system which, like all complex adaptive systems, can handle perturbations over the same timescale as the interal homostatic processes. Beyond that regime, the system will not adapt. If I tap your head, you're fine. If you knock you with an anvil, you're dead.
This wouldn't work. Wawaluigis are not luigis.
I believe we should limit AI development to below 0.2 OOMs/year which is slow continuous takeoff.
I've added a section on hardware:
- Comparing 0.2 OOMs/year target to hardware growth-rates:
- Moore's Law states that transitiors per integrated circuit doubles roughly every 2 years.
- Koomey's Law states that the FLOPs-per-Joule doubled roughly every 1.57 years until 2000, whereupon it began doubling roughly every 2.6 years.
- Huang's Law states that the growth-rate of GPU performance exceeds that of CPU performance. This is a somewhat dubious claim, but nonetheless I think the doubling time of GPUs is longer than 18 months.
- In general, the 0.2 OOMs/year target is faster than the current hardware growth-rate.
I think you've misunderstood what we mean by "target". Similar issues applied to the 2°C target, which nonetheless yielded significant coordination benefits.
The 2°C target helps facilitate coordination between nations, organisations, and individuals.
- It provided a clear, measurable goal.
- It provided a sense of urgency and severity.
- It promoted a sense of shared responsibility.
- It helped to align efforts across different stakeholders.
- It created a shared understanding of what success would look like.
The AI governance community should converge around a similar target.
- Nuclear proliferation worked despite the fact that many countries with nuclear weapons were "grandfathered in".
- If the y-axis for the constraint is fixed to the day of the negotiaiton, then stakeholders who want a laxer constraint are incentivised to delay negotiation. To avoid that hazard, I have picked a schelling date (2022) to fix the y-axis.
- The purpose of this article isn't to proposal any policy, strategy, treaty, agreement, law, etc which might achieve the 0.2 OOMs/year target. instead, the purpose of this article is to propose a target itself. This has inherent coordination benefits, c.f. the 2ºC target.
This isn't a policy proposal, it's a target, like the 2°C climate target.
Yep, thanks! 0.2 OOMs/year is equivalent to a doubling time of 18 months. I think that was just a typo.
The 0.2 OOMs/year target would be an effective moratorium until 2029, because GPT-4 overshot the target.
Yep, you're correct. The original argument in the Waluigi mega-post was sloppy.
- If updated the amplitudes in a perfectly bayesian way and the context window was infinite, then the amplitudes of each premise must be a martingale. But the finite context breaks this.
- Here is a toy model which shows how the finite context window leads to Waluigi Effect. Basically, the finite context window biases the Dynamic LLM towards premises which can be evidenced by short strings (e.g. waluigi), and biases away from premises which can't be evidenced by short strings (e.g. luigis).
- Regarding your other comment, a long context window doesn't mean that the waluigis won't appear quickly. Even with an infinite context window, the waluigi might appear immediately. The assumption that the context window is short/finite is only necessary to establish that the waluigi is an absorbing state but luigi isn't.
Yep, exactly!
Two things to note:
(1)
Note that the distinction between hinge beliefs and free beliefs does not supervene on the black-box behaviour of NNs/LLMs. It depends on how the belief is implemented, how the belief is learned, how the belief might change, etc.
(2)
"The second model uses a matrix that will always be symmetric, no matter what it's learned." might make it seem that the two models are more similar than they actually are.
You might think that both models store an matrix , and the architecture of both models is , but Model 1 has a slightly symmetric matrix whereas Model 2 has an exactly symmetric matrix . But this isn't true. The second model doesn't store a symmetric matrix — it stores an upper triangle.
The proposition "I am currently on Earth" is implemented both in the parameters and in the architecture, independently.
I think my definition of is correct. It's designed to abstract away all the messy implementation details of the ML architecture and ML training process.
Now, you can easily amend the definition to include an infinite context window . In fact, if you let then that's essentially an infinite context window. But it's unclear what optimal inference is supposed to look like when . When the context window is infinite (or very large) the internet corpus consists of a single datapoint.
Yep, but it's statistically unlikely. It is easier for order to disappear than for order to emerge.
I've spoken to some other people about Remark 1, and they also seem doubtful that token deletion is an important mechanism to think about, so I'm tempted to defer to you.
But on the inside view:
The finite context window is really important. 32K is close enough to infinity for most use-cases, but that's because users and orgs are currently underutilising the context window. The correct way to utilise the 32K context window is to fill it with any string of tokens which might help the computation.
Here's some fun things to fill the window with —
- A summary of facts about the user.
- A summary of news events that occurred after training ended. (This is kinda what Bing Sydney does — Bing fills the context window with the search results key words in the user's prompt.)
- A "compiler prompt" or "library prompt". This is a long string of text which elicits high-level, general-purpose, quality-assessed simulacra which can then be mentioned later by the users. (I don't mean simulacra in a woo manner.) Think of "import numpy as np" but for prompts.
- Literally anything other than null tokens.
I think no matter how big the window gets, someone will work out how to utilise it. The problem is that the context window has grown faster than prompt engineering, so no one has realised yet how to properly utilise the 32K window. Moreover, the orgs (Anthropic, OpenAI, Google) are too much "let's adjust the weights" (fine-tuning, RLHF, etc), rather than "let's change the prompts".
My guess is that the people voting "disagree" think that including the distillation in your general write-up is sufficient, and that you don't need to make the distillation its own post.
- Almost certainly ergodic in the limit. But it's highly period due to English grammar.
- Yep, just for convenience.
- Yep.
- Temp = 0 would give exactly absorbing states.
Maybe I should break this post down into different sections, because some of the remarks are about LLM Simulators, and some aren't.
Remarks about LLM Simulators: 7, 8, 9, 10, 12, 17
Other remarks : 1, 2, 3, 4, 5, 6, 11, 13, 14, 15, 16, 18
Yeah, I broadly agree.
My claim is that the deep metaphysical distinction is between "the computer is changing transistor voltages" and "the computer is multiplying matrices", not between "the computer is multiplying matrices" and "the computer is recognising dogs".
Once we move to a language game in which "the computer is multiplying matrices" is appropriate, then we are appealing to something like the X-Y Criterion for assessing these claims.
The sentences are more true the tighter the abstraction is —
- The machine does X with greater probability.
- The machine does X within a larger range of environments.
- The machine has fewer side effects.
- The machine is more robust to adversarial inputs.
- Etc
But SOTA image classifiers are better at recognising dogs than humans are, so I'm quite happy to say "this machine recognises dogs". Sure, you can generate adversarial inputs, but you could probably do that to a human brain as well if you had an upload.
We could easily train an AI to be 70 percentile in recognising human emotions, but (as far as I know) no one has bothered to do this because there is ~ 0 tangible benefit so it wouldn't justify the cost.
Recognising dogs by ML classification is different to recognising dogs using cells in your brain and eyes
Yeah, and the way that you recognise dogs is different from the way that cats recognise dogs. Doesn't seem to matter much.
as though it were exactly identical
Two processes don't need to be exactly identical to do the same thing. My calculator adds numbers, and I add numbers. Yet my calculator isn't the same as my brain.
when you invoke pop sci
Huh?
No it's not because one is sacred and the other is not, you've confused sacredness with varying degrees of complexity.
What notion of complexity do you mean? People are quite happy to accept that computers can perform tasks with high k-complexity or t-complexity. It is mostly "sacred" things (in the Hansonian sense) that people are unwilling to accept.
or you could continue to say AI feels things
Nowhere in this article to I address AI sentience.
I'm probably misunderstanding you but —
- A task is a particular transformation of the physical environment.
- COPY_POEM is the task which turns one page of poetry into two copies of the poetry.
The task COPY_POEM would be solved by a photocopier or a plagiarist schoolboy. - WRITE_POEM is the task which turns no pages of poetry into one page of poetry.
The task WRITE_POEM would be solved by Rilke or a creative schoolboy. - But the task COPY_POEM doesn't reduce to WRITE_POEM.
(You can imagine that although Rilke can write original poems, he is incapable of copying an arbitrary poem that you hand him.) - And the task WRITE_POEM doesn't reduce to COPY_POEM.
(My photocopier can't write poetry.)
I presume you mean something different by COPY_POEM and WRITE_POEM.
amended. thanks 🙏
There's no sense in which my computer is doing matrix multiplication but isn't recognising dogs.
At the level of internal mechanism, the computer is doing neither, it's just varying transistor voltages.
If you admit a computer can be multiplying matrices, or sorting integers, or scheduling events, etc — then you've already appealed to the X-Y Criterion.
I think the best explanation of why ChatGPT responds "Paris" when asked "What's the capital of France?" is that Paris is the capital of France.
Let's take LLM Simulator Theory.
We have a particular autoregressive language model , and Simulator Theory says that is simulating a whole series of simulacra which are consistent with the prompt.
Formally speaking,
where is the stochastic process corresponding to a simulacrum .
Now, there are two objections to this:
- Firstly, is it actually true that has this particular structure?
- Secondly, even if it were true, why are we warranted in saying that GPT is simulating all these simulacra?
The first objection is a purely technical question, whereas the second is conceptual. In this article, I present a criterion which partially answers the second objection.
Note that the first objection — is it actually true that has this particular structure? — is a question about a particular autoregressive language model. You might give one answer for GPT-2 and a different answer for GPT-4.
Yep, the problem is that the internet isn't written by Humans, so much as written by Humans + The Universe. Therefore, GPT-N isn't bounded by human capabilities.
We have two ontologies:
- Physics vs Computations
- State vs Information
- Machine vs Algorithm
- Dynamics vs Calculation
There's a bridge connecting these two ontologies called "encoding", but (as you note) this bridge seems arbitrary and philosophically messy. (I have a suspicion that this problem is mitigated if we consider quantum physics vs quantum computation, but I digress.)
This is why I don't propose that we think about computational reduction.
Instead, I propose that we think about physical reduction, because (1) it's less philosophically messy, (2) it's more relevant, and (3) it's more general.
We can ignore the "computational" ontology altogether. We don't need it. We can just think about expending physical resources instead.
- If I can physically interact with my phone (running Google Maps) to find my way home, then my phone is a route-finder.
- If I can use the desktop-running-Stockfish to win chess, then the desktop-running-Stockfish is a chess winner.
- If I can use the bucket and pebbles to count my sheep, then the bucket is a sheep counter.
- If I can use ChatGPT to write poetry, then ChatGPT is a poetry writer.
Yep, I broadly agree. But this would also apply to multiplying matrices.
Sure, every abstraction is leaky and if we move to extreme regimes then the abstraction will become leakier and leakier.
Does my desktop multiply matrices? Well, not when it's in the corona of the sun. And it can't add 10^200-digit numbers.
So what do we mean when we say "this desktop multiplies two matrices"?
We mean that in the range of normal physical environments (air pressure, room temperature, etc), the physical dynamics of the desktop corresponds to matrix multiplication with respect to some conventional encoding o small matrices into the physical states of the desktop
By adding similar disclaimers, I can say "this desktop writes music" or "this desktop recognises dogs".