LessWrong 2.0 Reader

It’s reasonable to consider two agents playing against each other. “Playing against your copy” is a reasonable problem. ($9 rocks get 0 in this problem, LDTs probably get $5.)

Newcomb, Parfit’s hitchhiker, smoking, etc. are all very reasonable problems that essentially depend on the buttons you press when you play the game. It is important to get these problems right.

But playing against LDT is not necessarily in the “fair problem class” because the game might behave differently depending on your algorithm/on how you arrive at taking actions, and not just depending on your actions.

Your version of it- playing against an LDT- is indeed different from playing against a game that looks at whether we’re an alphabetizing agent and pick X instead of Y because X<Y and not because we looked at the expected utility: we would want LDT to perform optimally in this game. But the reason LDT-created-rock loses to a natural rock here isn’t fundamentally different from the reason LDT loses to an alphabetizing agent in the other game and it is known that you can construct a game like that where LDT will lose to something else. You can make the game description sound more natural, but I feel like there’s a sharp divide between the “fair problem class” problems and others.

(I also think that in real life, where this game might play out, there isn’t really a choice we can make, to make our AI a $9 rock instead of an LDT agent; because when we do that due to the rock’s better performance in this game, our rock gets slightly less than $5 in EV instead of getting $9; LDT doesn’t perform worse than other agents we could’ve chosen in this game.)

Many states have already passed something like this, which only takes effect once enough states sign on. https://en.wikipedia.org/wiki/National_Popular_Vote_Interstate_Compact

However, I think it's unlikely that this will get over the 270 hump anytime soon, because right now GOP-run states (correctly) perceive that the EC has a pro-GOP tilt (for now at least); and the swing states benefit a lot from swing status.

On the other hand, that does not mean that those fields are going to recognize [a smart outsider] as a thought-leader or whatever.

Could this be a more general problem that people in academia are very unlikely to recognize anything that originates outside of academia?

I mean, first you get your degree by listening to people in academia, and reading textbooks written by people in academia. Then you learn to solve problems, defined by people in academia, under the leadership of people in academia, mostly by reading a lot of papers written and reviewed by people in academia. Etc. I think it is very easy to forget that the universe outside of academia exists at all.

And even if you remember that the universe outside of academia exists, interacting with it is potentially dangerous for your reputation. If you cite something from an academically respected source, then... even if it later turns out to be completely wrong, it kinda wasn't your mistake, was it? Compare to the situation where you cite something from a smart outsider... who also turns out to be completely wrong. Now you look like an idiot... and you can't shift the blame to him (because he is outside the system) nor to his reviewers (because either he had none, or they were also outside the system).

Similarly, suppose that two respected scientists have expressed a similar idea, and you prefer one of the formulations over the other. No problem; just cite the one you prefer. Now suppose that a respected scientist wrote something about a topic, but a smart outsider wrote about something slightly more relevant for your specific case and/or he expressed it better. Citing the outsider instead of the respected scientist... will make you seem like you are not familiar with the existing research in your field. (Chances are, the respected scientist or his colleague will review your paper.)

So it's part familiarity, part incentives.

Epistemic status: just a guess; I am not working in academia, so I only imagine what the incentives are

.

Now assuming that this is true, or at least pointing in the right direction, the problem seems solvable, but it would require some work. For example, you can't expect people in academia to go outside and read your research on their own initiative. But maybe if you publish it as a book that is easy to read, and donate them the book? Or maybe you could academia-wash your research by making someone inside the system a co-author.

Perhaps you could do both of these at the same time, like publish a popular science book that would contain articles both from LW researchers and academic researchers. And then the book would kinda be okay to cite.

This project seems to be trying to translate whale language.

Justification for this:

I don't think organisms end up with 40 billion cortical neurons without either some strong selection for at least some sub-dimensions of intelligence, or being as big as Godzilla.

One could naively expect that the neuron count (especially touch and motor) sensory processing modules are proportional to the surface area of an organism. However I think this is unrealistic: Bears don't need nearly as fine precision on what square centimeter of skin was touched (or what millimeter the paw moves) than mice, and generally this is because precision gets less relevant given body size.

So let's say the precision an organism needs is proportional to the square root of the 1-dimensional-size (aka sqrt(surface_area)) of the organism. Aka if a mice is 5cm tall and a bear 2m, the spacing between sensors on the mouse skin vs on the bear skin would be sqrt(0.05) vs sqrt(2). The number of sensors on the skin surface is proportional to the square of the distancing between sensors, so the overall number of sensors is proportional to the 1-dimensional-size (aka sqrt(surface_area)).

A brown bear has 250million neorons in the neocortex and is maybe 2m tall. So to get just by scaling size to 40billion neorons an organism would have to be 40/0.25 * 2m = 320m tall. So actually bigger than godzilla.

To prevent this, there must be a provision that once signed by all 4 states, the compact can't be repealed by any state until after the next election.

It's not obvious that state legislatures have the authority, under their own constitutions, to bind themselves that way. Especially not across their own election cycles.

LDT decision theories are probably the best decision theories for problems in the fair problem class.

The post demonstrates why this statement is misleading.

If "play the ultimatum game against a LDT agent" is not in the fair problem class, I'd say that LDT shouldn't be in the "fair agent class". It is like saying that in a tortoise-only race, the best racer is a hare because a hare can beat all the tortoises.

So based on the definitions you gave I'd classify "LDT is the best decision theory for problems in the fair problem class" as not even wrong.

In particular, consider a class of allowable problems S, but then also say that an agent X is allowable only if "play a given game with X" is in S. Then the proof in the No agent is rational in every problem section of my proof goes through for allowable agents. (Note that that argument in that section is general enough to apply to agents that don't give into $9 rock.)

Practically speaking: if you're trying to follow decision theory X, than playing against other X is a reasonable problem

+1 for just throwing your notes up on a website. For example, mine are at https://www.hgreer.com/Reports/ although there is currently a bit of a gap for the last few months as I've been working more on synthesizing existing work into a CVPR submission than on exploreing new directions.

The above is a terrible post-hoc justification and I need to get back to note taking.

The "sin, confession, priest" metaphor is a great anchor - that's very similar to the feeling we try to project when we have these discussions with clients.  And, related to Raemon's question, that's part of how I try to address the issue.

Specifically, we cast the instruction as much more of a simple commandment, leaving off the "But if you do" clause entirely from the particular guidance.  So in this case, the instruction is: "Don't reverse engineer competitor technology.  Including, don't buy copies of competitor tech or go searching for inside information.  This information is risky for us to have (for various legal reasons I can bore you with, but aren't important right now), and so you DON'T WANT IT."

This guidance, along with any other 'commandments' goes into the general body of "Legal says 'Don't'" instructions.  Which is more or less what most folks expect from a legal department.  (We work hard not to be, but a lot of legal teams are considered the "Department of 'No' ".)

And then, we overlay the number one rule over the top of all other guidance, and we finish with this every time: if you think you did something you shouldn't have, tell us right away.  If we don't know, we can't help you.

So to the original metaphor, we try to make ourselves part of the path to absolution (or at least mitigation), so that the rules can be simpler.

On the whole, it's a little bit of the reverse of the original "Don't ...  But ..." construction.  We try to flip it to: "We can help you if you screw up, but only if you tell us.  Here's what we hope we don't have to help you with."  This way, if they remember a "Don't", it's all good.  If they forget a "Don't", but they remember "in case of doubt, ask legal", we're mostly OK.  (If you sin, confess and get forgiven.  If you don't sin, good for you.)

In more semantic terms, we try to avoid the "But if you do ..." construction, since that seems to stick with people in a way that ignores the "Don't" part of it.  By making the top rule, "let me help you if you break a rule", it's easier to keep from making the activity that breaks the rule sound like a premise for positive action.

I wouldn't say we do great with this, but it's part of an overall effort to cast legal as useful advisors (also a priestly role), rather than enforcers. 

Now it's been four years of fast AI progress; I wonder if there are any updates? Has anyone tried to use machine learning to translate nonhuman communications?