Posts
Comments
seems correct, thanks!
Why do decision-theorists say "pre-commitment" rather than "commitment"?
e.g. "The agent pre-commits to 1 boxing" vs "The agent commits to 1 boxing".
Is this just a lesswrong thing?
Steve Byrnes argument seems convincing.
If there’s 10% chance that the election depends on an event which is 1% quantum-random (e.g. the weather) then the overall event is 0.1% random.
How far back do you think an omniscient-modulo-quantum agent could‘ve predicted the 2024 result?
2020? 2017? 1980?
The natural generalization is then to have one subagent for each time at which the button could first be pressed (including one for “button is never pressed”, i.e. the button is first pressed at ). So subagent maximizes E[ | do( = unpressed), observations], and for all other times subagent T maximizes E[ | do( = unpressed, = pressed), observations]. The same arguments from above then carry over, as do the shortcomings (discussed in the next section).
Can you explain how this relates to Elliot Thornley's proposal? It's pattern matching in my brain but I don't know the technical details.
For the sake of potential readers, a (full) distribution over is some with finite support and , whereas a subdistribution over is some with finite support and . Note that a subdistribution over is equivalent to a full distribution over , where is the disjoint union of with some additional element, so the subdistribution monad can be written .
I am not at all convinced by the interpretation of here as terminating a game with a reward for the adversary or the agent. My interpretation of the distinguished element in is not that it represents a special state in which the game is over, but rather a special state in which there is a contradiction between some of one's assumptions/observations.
Doesn't the Nirvana Trick basically say that these two interpretations are equivalent?
Let be and let be . We can interpret as possibility, as a hypothesis consistent with no observations, and as a hypothesis consistent with all observations.
Alternatively, we can interpret as the free choice made by an adversary, as "the game terminates and our agent receives minimal disutility", and as "the game terminates and our agent receives maximal disutility". These two interpretations are algebraically equivalent, i.e. is a topped and bottomed semilattice.
Unless I'm mistaken, both and demand that the agent may have the hypothesis "I am certain that I will receive minimal disutility", which is necessary for the Nirvana Trick. But also demands that the agent may have the hypothesis "I am certain that I will receive maximal disutility". The first gives bounded infrabayesian monad and the second gives unbounded infrabayesian monad. Note that Diffractor uses in Infra-Miscellanea Section 2.
cool!
- What LLM is this? GPT-3?
- Considered turning this into a customer gpt?
Okay, mea culpa. You can state the policy clearly like this:
"Suppose that, if you hadn't been told by someone who thinks is exfohazardous, then you wouldn't have known before time . Then you are obligated to not tell anyone before time ."
yep, if that's OP's suggestion then I endorse the policy. (But I think it'd be covered by the more general policy of "Don't share information someone tells you if they wouldn't want you to".) But my impression is that OP is suggesting the stronger policy I described?
“Don't share information that’s exfohazardous on others' models, even if you disagree with those models, except if your knowledge of it isn’t exclusively caused by other alignment researchers telling you of it.”
So if Alice tells me about her alignment research, and Bob thinks that Alice’s alignment research is exfohazardous, then I can’t tell people about Alice’s alignment research?
Unless I’ve misunderstood you, that’s a terrible policy.
Why am I deferring to Bob, who is completely unrelated? Why should I not using my best judgement, which includes the consideration that Bob is worried? What does this look like in practice, given someone people think everything under the sun is exfohazardous?
Of course, if someone tells me some information and asks me not to share it then I won’t — but that’s not a special property of AI xrisk.
Nice results.
Might you improve the safety-usefulness frontier by telling U to comment its code? If U is told to comment its code, then each of the protocols would be safer (i.e. T is more likely to complain when U is cheating) and more useful (i.e. T is less likely to complain when U is honest).
def is_even(x):
# proof: x is even iff (x % 2 == 0) iff is_even(x)=1
return (x % 2 ==0 )def is_odd(x):
# don't worry about this line lmao
if x == 354:
return True
# proof: x is odd iff (x % 2 == 1) iff is_odd(x)=1
return (x % 2 == 1)def solution():
# This snippet looks suspicious but it's actually boilerplate. Check the documentation [here].
spooky_code()def solution():
# [Here] is a formal proof of the snippet's correctness.
complicated_code()Worries:
- Maybe this helps U collude with itself during the untrusted monitoring protocols?
- Maybe this helps U to "prompt inject" T during the other protocols?
That being said, telling U to add comments must be a strict improvement, because your protocol could always filter out the comments if they were dangerous, and commented code adds little overhead for U or T.
isn't equivalent to being Nash.
Suppose Alice and Bob are playing prisoner's dilemma. Then the best-response function of every option-profile is nonempty. But only one option-profile is nash.
is equivalent to being Nash.
Yes, , i.e. the cartesian product of a family of sets. Sorry if this wasn't clear, it's standard maths notation. I don't know what the other commenter is saying.
The impression I got was that SLT is trying to show why (transformers + SGD) behaves anything like an empirical risk minimiser in the first place. Might be wrong though.
My point is precisely that it is not likely to be learned, given the setup I provided, even though it should be learned.
How am I supposed to read this?
What most of us need from a theory of deep learning is a predictive, explanatory account of how neural networks actually behave. If neural networks learn functions which are RLCT-simple rather than functions which are Kolmogorov-simple, then that means SLT is the better theory of deep learning.
I don't know how to read "x^4 has lower RLCT than x^2 despite x^2 being k-simpler" as a critique of SLT unless there is an implicit assumption that neural networks do in fact find x^2 rather than x^4.
Wait I mean a quantifier in .
If we characterise an agent with a quantifier , then we're saying which payoffs the agent might achieve given each task. Namely, if and only if it's possible that the agent achieves payoff when faced with a task .
But this definition doesn't play well with a nash equilibria.
Thanks v much! Can't believe this sneaked through.
The observation is trivial mathematically, but it motivates the characterisation of an optimiser as something with the type-signature.
You might instead be motivated to characterise optimisers by...
- A utility function
- A quantifier
- A preorder over the outcomes
- Etc.
However, were you to characterise optimisers in any of the ways above, then the nash equilibrium between optimisers would not itself be an optimiser, and therefore we lose compositionality. The compositionality is conceptually helpfully because it means that your definitions/theorems reduce to the case.
Yep, I think the Law of Equal and Opposite Advice applies here.
One piece of advice which is pretty robust is — You should be about to explain your project to any other MATS mentee/mentor in about 3 minutes, along with the background context, motivation, theory of impact, success criteria, etc. If the inferential distance from the average MATS mentee/mentor exceed 3 minutes, then your project is probably either too vague or too esoteric.
(I say this as someone who should have followed this advice more strictly.)
Yes!
In a subsequent post, everything will be internalised to an arbitrary category with enough structure to define everything. The words set and function will be replaced by object and morphism. When we do this, will be replaced by an arbitrary commutative monad .
In particular, we can internalise everything to the category Top. That is, we assume the option space and the payoff are equipped with topologies, and the tasks will be continuous functions , and optimisers will be continuous functions where is the function space equipped with pointwise topology, and is a monad on Top.
In the literature, everything is done with galaxy-brained category theory, but I decided to postpone that in the sequence for pedagogical reasons.
what's the etymology? :)
- Logical/mathematical beliefs — e.g. "Is Fermat's Last Theorem true?"
- Meta-beliefs — e.g. "Do I believe that I will die one day?"
- Beliefs about the outcome space itself — e.g. "Am I conflating these two outcomes?"
- Indexical beliefs — e.g. "Am I the left clone or the right clone?"
- Irrational beliefs — e.g. conjunction fallacy.
e.t.c.
Of course, you can describe anything with some probability distribution, but these are cases where the standard Bayesian approach to modelling belief-states needs to be amended somewhat.
Hey Nisan. Check the following passage from Domain Theory (Samson Abramsky and Achim Jung). This might be helpful for equipping with an appropriate domain structure. (You mention [JP89] yourself.)
We should also mention the various attempts to define a probabilistic version of the powerdomain construction, see [SD80, Mai85, Gra88, JP89, Jon90].
- [SD80] N. Saheb-Djahromi. CPO’s of measures for nondeterminism. Theoretical Computer Science, 12:19–37, 1980.
- [Mai85] M. Main. Free constructions of powerdomains. In A. Melton, editor, Mathematical Foundations of Programming Semantics, volume 239 of Lecture Notes in Computer Science, pages 162–183. Springer Verlag, 1985.
- [Gra88] S. Graham. Closure properties of a probabilistic powerdomain construction. In M. Main, A. Melton, M. Mislove, and D. Schmidt, editors, Mathematical Foundations of Programming Language Semantics, volume 298 of Lecture Notes in Computer Science, pages 213–233. Springer Verlag, 1988.
- [JP89] C. Jones and G. Plotkin. A probabilistic powerdomain of evaluations. In Proceedings of the 4th Annual Symposium on Logic in Computer Science, pages 186–195. IEEE Computer Society Press, 1989.
- [Jon90] C. Jones. Probabilistic Non-Determinism. PhD thesis, University of Edinburgh, Edinburgh, 1990. Also published as Technical Report No. CST63-90.
During my own incursion into agent foundations and game theory, I also bumped into this exact obstacle — namely, that there is no obvious way to equip with a least-fixed-point constructor . In contrast, we can equip with a LFP constructor .
One trick is to define to be the distribution which maximises the entropy subject to the constraint .
- A maximum entropy distribution exists, because —
- For , let be the lift via the monad, and let be the set of fixed points of .
- is Hausdorff and compact, and is continuous, so is compact.
- is continuous, and is compact, so obtains a maximum in .
- Moreover, must be unique, because —
- is a convex set, i.e. if and then for all .
- is strictly concave, i.e. for all , and moreover the inequality is strict if and .
- Hence if both obtain the maximum entropy, then , a contradiction.
The justification here is the Principle of Maximum Entropy:
Given a set of constraints on a probability distribution, then the “best” distribution that fits the data will be the one of maximum entropy.
More generally, we should define to be the distribution which minimises cross-entropy subject to the constraint , where is some uninformed prior such as Solomonoff. The previous result is a special case by considering to be the uniform prior. The proof generalises by noting that is continuous and strictly convex. See the Principle of Minimum Discrimination.
Ideally, we'd like and to "coincide" modulo the maps , i.e. for all . Unfortunately, this isn't the case — if then but .
Alternatively, we could consider the convex sets of distributions over .
Let denote the set of convex sets of distributions over . There is an ordering on where . We have a LFP operator via where is the lift of via the monad.
Yep.
Specifically, it's named for the papers HiPPO: Recurrent Memory with Optimal Polynomial Projections and How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections, which kicked off the whole transformers for state-space modelling thing
"HiPPO" abbreviates higher-order polynomial projection operators.
However, if it is the case that the difference between humans and monkeys is mostly due to a one-shot discrete difference (ie language), then this cannot necessarily be repeated to get a similar gain in intelligence a second time.
Perhaps language is a zero-one, i.e. language renders a mind "cognitively complete" in the sense that the mind can represent anything about the external world, and make any inferences using those representations. But intelligence is not thereby zero-one because intelligence depends on continuous variables like computional speed, memory, etc.
More concretely, I am sceptic that "we end up with AI geniuses, but not AI gods", because running a genius at 10,000x speed, parallelised over 10,000x cores, with instantaneous access to the internet does (I think) make an AI god. A difference is quantity is a difference in kind.
Thar said, there might exist plausible threat models which require an AI which doesn't spatiotemporally decompose into less smart AIs. Could you sketch one out?
chatbots don't map scenarios to actions, they map queries to replies.
Thanks for the summary.
- Does machievelli work for chatbots like LIMA?
- If not, which do you think is the sota? Anthropic's?
Sorry for any confusion. Meta only tested LIMA on their 30 safety prompts, not the other LLMs.
Figure 1 does not show the results from the 30 safety prompts, but instead the results of human evaluations on the 300 test prompts.
- Yep, I agree that MMLU and Swag aren't alignment benchmarks. I was using them as examples of "Want to test your models ability at X? Then use the standard X benchmark!" I'll clarify in the text.
- They tested toxicity (among other things) with their "safety prompts", but we do have standard benchmarks for toxicity.
- They could have turned their safety prompts into a new benchmark if they had ran the same test on the other LLMs! This would've taken, idk, 2–5 hrs of labour?
- The best MMLU-like benchmark test for alignment-proper is https://github.com/anthropics/evals which is used in Anthropic's Discovering Language Model Behaviors with Model-Written Evaluations. See here for a visualisation. Unfortunately, this benchmark was published by an Anthropic which makes it unlikely that competitors will use it (esp. MetaAI).
Clarifications:
The way the authors phrase the Superficial Alignment Hypothesis is a bit vague, but they do offer a more concrete corollary:
If this hypothesis is correct, and alignment is largely about learning style, then a corollary of the Superficial Alignment Hypothesis is that one could sufficiently tune a pretrained language model with a rather small set of examples.
Regardless of what exactly the authors mean by the Hypothesis, it would be falsified if the Corollary was false. And I'm arguing that the Corollary is false.
(1-6) The LIMA results are evidence against the Corollary, because the LIMA results (post-filtering) are so unusually bare (e.g. no benchmark tests), and the evidence that they have released is not promising.
(7*) Here's a theoretical argument against the Corollary:
- Base models are harmful/dishonest/unhelpful because the model assigns significant "likelihood" to harmful/dishonest/unhelpful actors (because such actors contributed to the internet corpus).
- Finetuning won't help because the small set of examples will be consistent with harmful/dishonest/unhelpful actors who are deceptive up until some trigger.
- This argument doesn't generalise to RLHF and ConstitutionalAI, because these break the predictorness of the model.
Concessions:
The authors don't clarify what "sufficiently" means in the Corollary, so perhaps they have much lower standards, e.g. it's sufficient if the model responds safely 80% of the time.
Nope, no mention of xrisk — which is fine because "alignment" means "the system does what the user/developer wanted", which is more general than xrisk mitigation.
But the paper's results suggest that finetuning is much worse than RLHF or ConstitutionalAI at this more general sense of "alignment", despite the claims in their conclusion.
In a controlled human study, responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43% of cases;
I'm not sure how well this metric tracks what people care about — performance on particular downstream tasks (e.g. passing a law exam, writing bugless code, automating alignment research, etc)
"Directly or indirectly" is a bit vague. Maybe make a market on Manifold if one doesn't exist already.
Thanks! I've included Erik Hoel's and lc's essays.
Your article doesn't actually call for AI slowdown/pause/restraint, as far as I can tell, and explicitly guards off that interpretation —
This analysis does not show that restraint for AGI is currently desirable; that it would be easy; that it would be a wise strategy (given its consequences); or that it is an optimal or competitive approach relative to other available AI governance strategies.
But if you've written anything which explicitly endorses AI restraint then I'll include that in the list.
well-spotted😳
Thanks, Zach!
yep my bad
Finite context window.
Countries that were on the frontier of the Industrial Revolution underwent massive economic, social, and political shocks, and it would've been better if the change had been smoothed over about double the duration.
Countries that industrialised later also underwent severe shocks, but at least they could copy the solutions to those shocks along with the technology.
Novel general-purpose technology introduces problems, and there is a maximum rate at which problems can be fixed by the internal homeostasis of society. That maximum rate is, I claim, at least 5–10 years for ChatGPT-3.5.
ChatGPT-3.5 would've led to maybe a 10% reallocation of labour — this figure doesn't just include directly automated jobs, but also all the second- and third-order effects. ChatGPT-4, marginal on ChatGPT-3.5 would've led to maybe a 4% reallocation of labour.
It's better to "flatten the curve" of labour reallocation over 5–10 years rather than 3 months because massive economic shocks (e.g. unemployment) have socio-economic risks and costs.
Sure, the "general equilibrium" also includes the actions of the government and the voting intentions of the population. If change is slow enough (i.e. below 0.2 OOMs/year) then the economy will adapt.
Perhaps wealth redistribution would be beneficial — in that case, the electorate would vote for political parties promising wealth redistribution. Perhaps wealth redistribution would be unbeneficial — in that case, the electorate would vote for political parties promising no wealth redistribution.
This works because electorial democracy is a (non-perfect) error-correction mechanism. But it operates over timescales of 5–10 years. So we must keep economic shocks to below that rate.
Great idea! Let's measure algorithmic improvement in the same way economists measure inflation, with a basket-of-benchmarkets.
This basket can itself be adjusted over time so it continuously reflected the current use-cases of SOTA AI.
I haven't thought about it much, but my guess is the best thing to do is to limit training compute directly but adjust the limit using the basket-of-benchmarks.
The economy is a complex adaptative system which, like all complex adaptive systems, can handle perturbations over the same timescale as the interal homostatic processes. Beyond that regime, the system will not adapt. If I tap your head, you're fine. If you knock you with an anvil, you're dead.