LessWrong 2.0 Reader

+1 for just throwing your notes up on a website. For example, mine are at https://www.hgreer.com/Reports/ although there is currently a bit of a gap for the last few months as I've been working more on synthesizing existing work into a CVPR submission than on exploreing new directions.

The above is a terrible post-hoc justification and I need to get back to note taking.

The "sin, confession, priest" metaphor is a great anchor - that's very similar to the feeling we try to project when we have these discussions with clients.  And, related to Raemon's question, that's part of how I try to address the issue.

Specifically, we cast the instruction as much more of a simple commandment, leaving off the "But if you do" clause entirely from the particular guidance.  So in this case, the instruction is: "Don't reverse engineer competitor technology.  Including, don't buy copies of competitor tech or go searching for inside information.  This information is risky for us to have (for various legal reasons I can bore you with, but aren't important right now), and so you DON'T WANT IT."

This guidance, along with any other 'commandments' goes into the general body of "Legal says 'Don't'" instructions.  Which is more or less what most folks expect from a legal department.  (We work hard not to be, but a lot of legal teams are considered the "Department of 'No' ".)

And then, we overlay the number one rule over the top of all other guidance, and we finish with this every time: if you think you did something you shouldn't have, tell us right away.  If we don't know, we can't help you.

So to the original metaphor, we try to make ourselves part of the path to absolution (or at least mitigation), so that the rules can be simpler.

On the whole, it's a little bit of the reverse of the original "Don't ...  But ..." construction.  We try to flip it to: "We can help you if you screw up, but only if you tell us.  Here's what we hope we don't have to help you with."  This way, if they remember a "Don't", it's all good.  If they forget a "Don't", but they remember "in case of doubt, ask legal", we're mostly OK.  (If you sin, confess and get forgiven.  If you don't sin, good for you.)

In more semantic terms, we try to avoid the "But if you do ..." construction, since that seems to stick with people in a way that ignores the "Don't" part of it.  By making the top rule, "let me help you if you break a rule", it's easier to keep from making the activity that breaks the rule sound like a premise for positive action.

I wouldn't say we do great with this, but it's part of an overall effort to cast legal as useful advisors (also a priestly role), rather than enforcers. 

Now it's been four years of fast AI progress; I wonder if there are any updates? Has anyone tried to use machine learning to translate nonhuman communications?

See Deep Learning Systems Are Not Less Interpretable Than Logic/Probability/Etc [LW · GW], including my comment on it [LW(p) · GW(p)]. If your approach would lead to a world-model that is an uninterpretable inscrutable mess, and LLM research would lead to a world-model that is an even more uninterpretable, even more inscrutable mess, then I don’t think this is a reason to push forward on your approach, without a good alignment plan.

Yes, it’s a pro tanto reason to prefer your approach, other things equal. But it’s a very minor reason. And other things are not equal. On the contrary, there are a bunch of important considerations plausibly pushing in the opposite direction:

• Maybe LLMs will plateau anyway [LW · GW], so the comparison between inscrutable versus even-more-inscrutable is a moot point. And then you’re just doing AGI capabilities research for no safety benefit at all. (See “Endgame safety” for AGI [LW · GW].)
• LLMs at least arguably have some safety benefits related to reliance on human knowledge, human concepts, and chains-of-thought, whereas the kind of AGI you’re trying to invent might not have those.
• Your approach would (if “successful”) be much, much more compute-efficient—probably by orders of magnitude—see Section 3 here [LW · GW] for a detailed explanation of why. This is bad because, if AGI is very compute-efficient, then when we have AGI at all, we will have AGI that a great many actors around the world will be able to program and run, and that makes governance very much harder. (Related: I for one think AGI is possible on a single consumer GPU, see here [LW · GW].)
• Likewise, your approach would (if “successful”) have a “better” inductive bias, “better” sample efficiency, etc., because you’re constraining the search space. That suggests fast takeoff and less likelihood of a long duration of janky mediocre-human-level AGIs. I think most people would see that as net bad for safety.

In any case, it seems that this is a problem that any possible way to build an intelligence runs into? So I don't think it is a case against the project.

If it’s a problem for any possible approach to building AGI, then it’s an argument against pursuing any kind of AGI capabilities research! Yes! It means we should focus first on solving that problem, and only do AGI capabilities research when and if we succeed. And that’s what I believe. Right?

It seems plausible that one could simply by understanding the system very well, make it such that the learned datastrucutres need to take particular shapes, such that these shapes correspond some relevant alignment properties.

I don’t think this is plausible. I think alignment properties are pretty unrelated to the low-level structure out of which a world-model is built. For example, the difference between “advising a human” versus “manipulating a human”, and the difference between “finding a great out-of-the-box solution” versus “reward hacking”, are both extremely important for alignment. But you won’t get insight into those distinctions, or how to ensure them in an AGI, by thinking about whether world-model stuff is stored as connections on graphs versus induction heads or whatever.

Anyway, if your suggestion is true, I claim you can (and should) figure that out without doing AGI capabilities research. Here’s an example. Assume that the the learned data structure is a Bayes net, or some generalization of a Bayes net or whatever. OK, now spend as long as you like thinking about what if anything that has to do with “alignment properties”. My guess is “very little”. Or if you come up with anything, you can share it. You don’t need to figure out how an AI would built that Bayes net.

If you want something more concrete to chew on, I humbly suggest brain-like AGI [? · GW]. :)  We already know a lot about how it works without needing to do any new capabilities research. For example, you might start with Plan for mediocre alignment of brain-like [model-based RL] AGI [LW · GW] and think about how to make that approach better / less bad.

I agree with the thrust and most of the content of the post, but in the interest of strengthening it, I'm looking at your list of problems and wanted to point out what I see as gaps/weaknesses.

For the first one, keep in mind it took centuries from trying to develop a temperature scale to actually having the modern thermodynamic definition of temperature, and reliable thermometers. The definition is kinda weird and unintuitive, and strictly speaking runs from 0 to infinity, then discontinuously jumps to negative infinity (but only for some kinds of finite systems), then rises back towards negative zero (I always found this funny when playing the Sims 3 since it had a" -1K Refrigerator"). Humans knew things got hot and cold for many, many millennia before figuring out temperature in a principled way. Morality could plausibly be similar.

The third and fourth seem easily explainable by bounded rationality, in the same way that "ability to build flying machines and quantum computers" and "ability to identify and explain the fundamental laws of physical reality" vary between individuals, cultures, and societies.

For the fifth, there's no theoretical requirement that something real should only have a small number of principles that are necessary for human-scale application. Occam's Razor cuts against anyone suggesting a fundamentally complex thing, but it is possible there is a simple underlying set of principles that is just incredibly complicated to use in practice. I would argue that most attempts to formalize morality, from Kant to Bentham etc., have this problem, and one of the common ways they go wrong is that people try to apply them without recognizing that.

The sixth seems like a complete non-sequitur to me. If moral realism were true, then people should be morally good. But why would they? Even if there were somehow a satisfying answer to the second problem of imposing an obligation, this does not necessarily provide an actual mechanism to compel action or a trend to action to fulfil the obligation. In fact at least some traditional attempts to have moral realist frameworks, like Judeo-Christian God-as-Lawgiver religion, explicitly avoid having such a mechanism. 

A doctor being accountable to a director seems like it could work in both ways. Even both ways in parallel; the director could call out some kinds of problems and cause some different kinds of problems.

The actual outcome could depend on the specific doctor and the specific director, and therefore be difficult to predict. It could even change in time, like maybe at first the directors would behave a certain way to attract more doctors, and when the customers learn that the big centers are the norm, then it would become the right time to squeeze more profit out of the system.

Really interesting work, especially the three stages you find. Regarding your question whether your classifier will generalize to negated statements: This has been explored in our Neurips 2024 paper "Truth is Universal: Robust Detection of Lies in LLMs" (https://arxiv.org/abs/2407.12831). In fact, true and false negated statements separate along a different direction than statements without negation, so a classifier would not generalize. The truthfulness representation found in this paper is also universal and can be found in multiple LLMs which is consistent with your findings :) 

So why don't the four states sign a compact to assign all their electoral votes in 2028 and future presidential elections to the winner of the aggregate popular vote in those four states? Would this even be legal?

It would be legal to make an agreement like this (states are authorized to appoint electors and direct their votes however they like; see Chiafalo v. Washington) but it's not enforceable in the sense that if one of the states reneges, the outcome of the presidential election won't be reversed.

The following is probably just an ELI5 version of Dacyn's answer [LW(p) · GW(p)]:

Just because you use a word (such as "set"), it doesn't mean that it has an unambiguous meaning.

Imagine the same discussion about "numbers". Can you subtract 5 from 3? In the universe of natural numbers, the answer is no. In the universe of all integers, the answer is yes. Is there a number such that if you multiply it by itself, the result is 2? In the universe of rational numbers, the answer is no; in the universe of real numbers, the answer is yes.

Here you probably don't see any problem. Some statements can be true about real numbers and false about rational numbers, because those are two different things. A person who talks about "numbers" in general, needs to be more specific. As we see here, defining addition, subtraction, multiplication, and division is still not enough to allow us to figure out the answer to "∃a: a × a = 2".

It's similar with the sets. The ZF(C) axioms are simply not enough to pinpoint what you actually mean by a "set". They reduce the space of possible meanings, sufficiently to let us prove many interesting things, but there are still (infinitely) many possible meanings compatible with all of the axioms. For some of those meanings, CH is true, for other meanings, CH is false.

Is there a "set" greater than ℵ₀ but smaller than 2^ℵ₀? It depends.

Is there a "number" that is greater than 2 but smaller than 3? It depends.

What makes certain axioms "true" beyond mere consistency?

Nothing. What do you mean by "true" here? Matching our physical universe? That in general is not what math does. The natural numbers may already include some that exceed the number of particles in our universe. The real numbers are inspired by measuring actual things, but do we really need an infinite number of decimal places? On top of all that, sets are merely mental constructs. A set of {2, 8, 33897798} does not imply anything about our world.

Is there a meaningful distinction between mathematical existence and consistency?

No.

Can we maintain mathematical realism while acknowledging the practical utility of the multiverse approach?

Maybe I am missing some important aspect, but the "multiverse" seems to me just like a intuitively helpful metaphor, but the actual problem is more like this: is the natural number "2" the same object as the integer "2", the real number "2.0", the Gaussian integer "2+0i", the complex number "2.0+0.0i", etc.?

One possible approach is to say: those are different domains of discourse... uhm, let's call them parallel universes to make it intuitive for the sci-fi fans. The object in a parallel universe is a different object, but also in some sense the captain Picard from the parallel universe is a natural counterpart to our captain Picard. They are generally the same unless specified otherwise for the plot relevant reasons, just like "2.0" from the real number universe is the natural counterpart to "2" from the integer universe, except that the former can be divided by three and the latter cannot. (Some things do not have a counterpart in the other universe, etc.) This feels like a natural approach for real vs complex numbers, and probably like an overkill for natural numbers vs integers.

The assumption of different universes kinda goes against the Occam's razor; we could simply move all these objects into the same universe (different planets perhaps) and make a story about a spaceship captain from Earth and a spaceship captain from Mars. Now we don't have the concept of a natural counterpart, and the analogies need to make explicitly: the horses on Earth correspond to the giant six-legged lizards on Mars. There is a set of natural numbers, the set of real numbers, and a function N -> R which maps the object "2" to the object "2.0". More importantly, there is no such thing as "addition"; there are actually two different things, "natural number addition" and "real number addition", and we call the latter the extension of the former, if for each pair of natural numbers, the counterpart of their sum is the same as the sum of their counterparts. The question whether "2" and "2.0" are intrinsically the same object can become kinda meaningless, if we always talk about numbers qua members of one or the other set. They could be the same object, or they could be different objects; the important thing is what they do, i.e. how they participate in various functions and relations.

(This kinda reminds me of Korzybski's "Aristotelian" vs "Non-Aristotelian" thinking, where the former is about what things are, while the latter is about how things are related to each other. Is "2" the same as "2.0"? A meaningless question, from the non-A perspective. The important thing is what they do; how are they related to other numbers. The important facts about "2" are that "1+1=2" and "2+2=4" etc. We can show that we can map N to R in a way that preserves all existing addition and multiplication, and whenever we do so, "2.0" is the image of "2". And that's all there is.)

With sets, I guess it is similar. If we have different definitions of what a "set" means, is the empty set according to definition X the same mathematical object as the empty set according to definition Y? The question is meaningless, from the non-A perspective; but to avoid all the complicated philosophy, it is easier to say that one lives in the universe X, and the other lives in the universe Y, so they are "kinda the same, but not the same". But to be precise, there is no such thing as an "empty set", only something that plays the role of an empty set in a certain system. Some systems could not even have such role, or they could have multiple distinct empty sets -- for example, we could imagine a system where each set has a type, and the "empty set of integers" is different from the "empty set of reals", because it has a different content type.

(Now I suspect I have opened a new can of worms, like how to reconcile Platonism with Korzybski's non-A thinking, and... that would be a long debate that I would prefer to avoid. My quick opinion is that perhaps we should aim for some kind of "Platonism of function" rather than "Platonism of essence", i.e. what the abstract objects do rather than what they are. The question is whether we should still call this approach "Platonism", perhaps some other name would be better.)

Playing ultimatum game against an agent that gives in to $9 from rocks but not from us is not in the fair problem class, as the payoffs depend directly on our algorithm and not just on our choices and policies.

https://arbital.com/p/fair_problem_class/

A simpler game is “if you implement or have ever implemented LDT, you get $0; otherwise, you get $100”.

LDT decision theories are probably the best decision theories for problems in the fair problem class.

(Very cool that you’ve arrived at the idea of this post independently!)