LessWrong 2.0 Reader

On the other hand, that does not mean that those fields are going to recognize [a smart outsider] as a thought-leader or whatever.

Could this be a more general problem that people in academia are very unlikely to recognize anything that originates outside of academia?

I mean, first you get your degree by listening to people in academia, and reading textbooks written by people in academia. Then you learn to solve problems, defined by people in academia, under the leadership of people in academia, mostly by reading a lot of papers written and reviewed by people in academia. Etc. I think it is very easy to forget that the universe outside of academia exists at all.

And even if you remember that the universe outside of academia exists, interacting with it is potentially dangerous for your reputation. If you cite something from an academically respected source, then... even if it later turns out to be completely wrong, it kinda wasn't your mistake, was it? Compare to the situation where you cite something from a smart outsider... who also turns out to be completely wrong. Now you look like an idiot... and you can't shift the blame to him (because he is outside the system) nor to his reviewers (because either he had none, or they were also outside the system).

Similarly, suppose that two respected scientists have expressed a similar idea, and you prefer one of the formulations over the other. No problem; just cite the one you prefer. Now suppose that a respected scientist wrote something about a topic, but a smart outsider wrote about something slightly more relevant for your specific case and/or he expressed it better. Citing the outsider instead of the respected scientist... will make you seem like you are not familiar with the existing research in your field. (Chances are, the respected scientist or his colleague will review your paper.)

So it's part familiarity, part incentives.

Epistemic status: just a guess; I am not working in academia, so I only imagine what the incentives are

.

Now assuming that this is true, or at least pointing in the right direction, the problem seems solvable, but it would require some work. For example, you can't expect people in academia to go outside and read your research on their own initiative. But maybe if you publish it as a book that is easy to read, and donate them the book? Or maybe you could academia-wash your research by making someone inside the system a co-author.

Perhaps you could do both of these at the same time, like publish a popular science book that would contain articles both from LW researchers and academic researchers. And then the book would kinda be okay to cite.

This project seems to be trying to translate whale language.

Justification for this:

I don't think organisms end up with 40 billion cortical neurons without either some strong selection for at least some sub-dimensions of intelligence, or being as big as Godzilla.

One could naively expect that the neuron count (especially touch and motor) sensory processing modules are proportional to the surface area of an organism. However I think this is unrealistic: Bears don't need nearly as fine precision on what square centimeter of skin was touched (or what millimeter the paw moves) than mice, and generally this is because precision gets less relevant given body size.

So let's say the precision an organism needs is proportional to the square root of the 1-dimensional-size (aka sqrt(surface_area)) of the organism. Aka if a mice is 5cm tall and a bear 2m, the spacing between sensors on the mouse skin vs on the bear skin would be sqrt(0.05) vs sqrt(2). The number of sensors on the skin surface is proportional to the square of the distancing between sensors, so the overall number of sensors is proportional to the 1-dimensional-size (aka sqrt(surface_area)).

A brown bear has 250million neorons in the neocortex and is maybe 2m tall. So to get just by scaling size to 40billion neorons an organism would have to be 40/0.25 * 2m = 320m tall. So actually bigger than godzilla.

To prevent this, there must be a provision that once signed by all 4 states, the compact can't be repealed by any state until after the next election.

It's not obvious that state legislatures have the authority, under their own constitutions, to bind themselves that way. Especially not across their own election cycles.

LDT decision theories are probably the best decision theories for problems in the fair problem class.

The post demonstrates why this statement is misleading.

If "play the ultimatum game against a LDT agent" is not in the fair problem class, I'd say that LDT shouldn't be in the "fair agent class". It is like saying that in a tortoise-only race, the best racer is a hare because a hare can beat all the tortoises.

So based on the definitions you gave I'd classify "LDT is the best decision theory for problems in the fair problem class" as not even wrong.

In particular, consider a class of allowable problems S, but then also say that an agent X is allowable only if "play a given game with X" is in S. Then the proof in the No agent is rational in every problem section of my proof goes through for allowable agents. (Note that that argument in that section is general enough to apply to agents that don't give into $9 rock.)

Practically speaking: if you're trying to follow decision theory X, than playing against other X is a reasonable problem

+1 for just throwing your notes up on a website. For example, mine are at https://www.hgreer.com/Reports/ although there is currently a bit of a gap for the last few months as I've been working more on synthesizing existing work into a CVPR submission than on exploreing new directions.

The above is a terrible post-hoc justification and I need to get back to note taking.

The "sin, confession, priest" metaphor is a great anchor - that's very similar to the feeling we try to project when we have these discussions with clients.  And, related to Raemon's question, that's part of how I try to address the issue.

Specifically, we cast the instruction as much more of a simple commandment, leaving off the "But if you do" clause entirely from the particular guidance.  So in this case, the instruction is: "Don't reverse engineer competitor technology.  Including, don't buy copies of competitor tech or go searching for inside information.  This information is risky for us to have (for various legal reasons I can bore you with, but aren't important right now), and so you DON'T WANT IT."

This guidance, along with any other 'commandments' goes into the general body of "Legal says 'Don't'" instructions.  Which is more or less what most folks expect from a legal department.  (We work hard not to be, but a lot of legal teams are considered the "Department of 'No' ".)

And then, we overlay the number one rule over the top of all other guidance, and we finish with this every time: if you think you did something you shouldn't have, tell us right away.  If we don't know, we can't help you.

So to the original metaphor, we try to make ourselves part of the path to absolution (or at least mitigation), so that the rules can be simpler.

On the whole, it's a little bit of the reverse of the original "Don't ...  But ..." construction.  We try to flip it to: "We can help you if you screw up, but only if you tell us.  Here's what we hope we don't have to help you with."  This way, if they remember a "Don't", it's all good.  If they forget a "Don't", but they remember "in case of doubt, ask legal", we're mostly OK.  (If you sin, confess and get forgiven.  If you don't sin, good for you.)

In more semantic terms, we try to avoid the "But if you do ..." construction, since that seems to stick with people in a way that ignores the "Don't" part of it.  By making the top rule, "let me help you if you break a rule", it's easier to keep from making the activity that breaks the rule sound like a premise for positive action.

I wouldn't say we do great with this, but it's part of an overall effort to cast legal as useful advisors (also a priestly role), rather than enforcers. 

Now it's been four years of fast AI progress; I wonder if there are any updates? Has anyone tried to use machine learning to translate nonhuman communications?

See Deep Learning Systems Are Not Less Interpretable Than Logic/Probability/Etc [LW · GW], including my comment on it [LW(p) · GW(p)]. If your approach would lead to a world-model that is an uninterpretable inscrutable mess, and LLM research would lead to a world-model that is an even more uninterpretable, even more inscrutable mess, then I don’t think this is a reason to push forward on your approach, without a good alignment plan.

Yes, it’s a pro tanto reason to prefer your approach, other things equal. But it’s a very minor reason. And other things are not equal. On the contrary, there are a bunch of important considerations plausibly pushing in the opposite direction:

• Maybe LLMs will plateau anyway [LW · GW], so the comparison between inscrutable versus even-more-inscrutable is a moot point. And then you’re just doing AGI capabilities research for no safety benefit at all. (See “Endgame safety” for AGI [LW · GW].)
• LLMs at least arguably have some safety benefits related to reliance on human knowledge, human concepts, and chains-of-thought, whereas the kind of AGI you’re trying to invent might not have those.
• Your approach would (if “successful”) be much, much more compute-efficient—probably by orders of magnitude—see Section 3 here [LW · GW] for a detailed explanation of why. This is bad because, if AGI is very compute-efficient, then when we have AGI at all, we will have AGI that a great many actors around the world will be able to program and run, and that makes governance very much harder. (Related: I for one think AGI is possible on a single consumer GPU, see here [LW · GW].)
• Likewise, your approach would (if “successful”) have a “better” inductive bias, “better” sample efficiency, etc., because you’re constraining the search space. That suggests fast takeoff and less likelihood of a long duration of janky mediocre-human-level AGIs. I think most people would see that as net bad for safety.

In any case, it seems that this is a problem that any possible way to build an intelligence runs into? So I don't think it is a case against the project.

If it’s a problem for any possible approach to building AGI, then it’s an argument against pursuing any kind of AGI capabilities research! Yes! It means we should focus first on solving that problem, and only do AGI capabilities research when and if we succeed. And that’s what I believe. Right?

It seems plausible that one could simply by understanding the system very well, make it such that the learned datastrucutres need to take particular shapes, such that these shapes correspond some relevant alignment properties.

I don’t think this is plausible. I think alignment properties are pretty unrelated to the low-level structure out of which a world-model is built. For example, the difference between “advising a human” versus “manipulating a human”, and the difference between “finding a great out-of-the-box solution” versus “reward hacking”, are both extremely important for alignment. But you won’t get insight into those distinctions, or how to ensure them in an AGI, by thinking about whether world-model stuff is stored as connections on graphs versus induction heads or whatever.

Anyway, if your suggestion is true, I claim you can (and should) figure that out without doing AGI capabilities research. Here’s an example. Assume that the the learned data structure is a Bayes net, or some generalization of a Bayes net or whatever. OK, now spend as long as you like thinking about what if anything that has to do with “alignment properties”. My guess is “very little”. Or if you come up with anything, you can share it. You don’t need to figure out how an AI would built that Bayes net.

If you want something more concrete to chew on, I humbly suggest brain-like AGI [? · GW]. :)  We already know a lot about how it works without needing to do any new capabilities research. For example, you might start with Plan for mediocre alignment of brain-like [model-based RL] AGI [LW · GW] and think about how to make that approach better / less bad.

I agree with the thrust and most of the content of the post, but in the interest of strengthening it, I'm looking at your list of problems and wanted to point out what I see as gaps/weaknesses.

For the first one, keep in mind it took centuries from trying to develop a temperature scale to actually having the modern thermodynamic definition of temperature, and reliable thermometers. The definition is kinda weird and unintuitive, and strictly speaking runs from 0 to infinity, then discontinuously jumps to negative infinity (but only for some kinds of finite systems), then rises back towards negative zero (I always found this funny when playing the Sims 3 since it had a" -1K Refrigerator"). Humans knew things got hot and cold for many, many millennia before figuring out temperature in a principled way. Morality could plausibly be similar.

The third and fourth seem easily explainable by bounded rationality, in the same way that "ability to build flying machines and quantum computers" and "ability to identify and explain the fundamental laws of physical reality" vary between individuals, cultures, and societies.

For the fifth, there's no theoretical requirement that something real should only have a small number of principles that are necessary for human-scale application. Occam's Razor cuts against anyone suggesting a fundamentally complex thing, but it is possible there is a simple underlying set of principles that is just incredibly complicated to use in practice. I would argue that most attempts to formalize morality, from Kant to Bentham etc., have this problem, and one of the common ways they go wrong is that people try to apply them without recognizing that.

The sixth seems like a complete non-sequitur to me. If moral realism were true, then people should be morally good. But why would they? Even if there were somehow a satisfying answer to the second problem of imposing an obligation, this does not necessarily provide an actual mechanism to compel action or a trend to action to fulfil the obligation. In fact at least some traditional attempts to have moral realist frameworks, like Judeo-Christian God-as-Lawgiver religion, explicitly avoid having such a mechanism.