LessWrong 2.0 Reader

After reading your and FlorianH [LW(p) · GW(p)]'s comments, it seems to me that Econ 101 leaves it underspecified what it means to be an economical agent, and that those parts missing from the specification are the ones that matter here.

Naively, an economical agent is someone who accepts deals that increase the value they get. There seems to be nothing wrong with that; if we all become economical agents, our values will increase, which is a good thing.

But this is not the entire story. No agent accepts all hypothetical deals that would increase their value. Our attention and time are limited. We pick the seemingly best deals we are aware of. And there are probably other heuristics that successful agents follow, such as increasing their power, even if it does not increase the value in short term, because it will allow them to take more value in future.

People who insist on taking the Econ 101 perspective of "if the deal is not good for you, then simply don't take it, duh" seem willfully blind to how the power is strategically gained and used.

This reminds me of Ayn Rand's novels. Both the heroes and the villains could be called "economical agents" from certain perspective, but clearly they used different strategies. There is a difference between someone trying to get good deals without simultaneously crippling their trade partners, and someone for whom crippling their trade partners is an important component of how they get the good deals. Both of them are agents participating in the economy.

I think there is an obvious signal that could be used: a forecast of how much MIRI will like the research when asked in 5 years. (Note that I don't mean just asking MIRI now, but rather something like prediction markets or super-forecasters to predict what MIRI will say 5 years from now.)

Basically, if the forecast is above average, anyone who trusts MIRI should fund them.

Re the OpenAI o-series and search, my initial prediction is that Q*/MCTS search will work well on problems that are easy to verify and and easy to get training data for, and not work if either of these 2 conditions are violated, and secondarily will be reliant on the model having good error correction capabilities to use the search effectively, which is why I expect we can make RL capable of superhuman performance on mathematics/programming with some rather moderate schlep/drudge work, and I also expect cost reductions such that it can actually be practical, but I'm only giving a 50/50 chance by 2028 for superhuman performance as measured by benchmarks in these domains.

I think my main difference from you, Thane Ruthenis is I expect costs to reduce surprisingly rapidly, though this is admittedly untested.

This will accelerate AI progress, but not immediately cause an AI explosion, though in the more extreme paces this could create something like a scenario where programming companies are founded by a few people smartly managing a lot of programming AIs, and programming/mathematics experiencing something like what happened to the news industry from the rise of the internet, where there was a lot of bankruptcy of the middle end, the top end won big, and most people are in the bottom end.

Also, correct point on how a lot of people's conceptions of search are babble-and-prune, not top down search like MCTS/Q*/BFS/DFS/A* (not specifically targeted at sunwillrisee

By contrast, my understanding is that the sort of search John is talking about retargeting isn't the brute-force babble-and-prune algorithms, but a top-down heuristical-constraint-based search [LW · GW].
 

Maybe someone else could moderate it?

There's no good candidate for a simple, legible, easily-obtained, and agreeable-to-most metric.  Before-and-after polling of patients is probably closest we can get.

That said, the dimensions of quality that the FDA concerns itself with (including physical functioning, self-reported pain, and other easily- and not-easily-measured things) is likely close enough to "improves quality of life" that it's not necessary to have a new direction.  

Perhaps you could identify some drugs that you think would improve quality of life, and work backwards to the metrics that prove to you that they do so.

this concern sounds like someone walking down a straight road and then closing their eyes cause they know where they want to go anyway

This doesn't sound like a good analogy at all. A better analogy might be a stylized subway map compared to a geographically accurate one. Sometimes removing detail can make it easier to process.

A policeman sees a drunk man searching for something under a streetlight and asks what the drunk has lost. He says he lost his keys and they both look under the streetlight together. After a few minutes the policeman asks if he is sure he lost them here, and the drunk replies, no, and that he lost them in the park. The policeman asks why he is searching here, and the drunk replies, "this is where the light is".

I've always been sympathetic to the drunk in this story.  If the key is in the light, there is a chance of finding it.  If it is in the dark, he's not going to find it anyway so there isn't much point in looking there.

Given the current state of alignment research, I think it's fair to say that we don't know where the answer will come from.  I support The Plan [LW · GW] and I hope research continues on it.  But if I had to guess, alignment will not be solved via getting a bunch of physicists thinking about agent foundations.  It will be solved by someone who doesn't know better making a discovery they "wasn't supposed to work".

On an interesting side here a fun story about experts repeatedly failing to make an obvious-in-hindsight discovery because they "knew better".

Seems to me that there is always some friction, some lack of information, etc., so "this requires market failure by definition" basically means "this happens in the real world".

Agreed. Simply focusing on physics post-docs feels too narrow to me.

Then again, just as John has a particular idea of what good alignment research looks like, I have my own idea: I would lean towards recruiting folk with both a technical and a philosophical background. It's possible that my own idea is just as narrow.

You can buy bulk tamiflu powder here: https://www.selleckchem.com/products/oseltamivir-phosphate-Tamiflu.html And instructions for preparation are here: https://dph.illinois.gov/content/dam/soi/en/web/idph/files/publications/tami-flu-flyer-050316.pdf $300 for 13 doses, which would last you about 6 days, dosing the recommended twice daily during a pandemic. Shelf life of the powder is 2 years. Would any medicine nerd sanity check that I am not missing something essential?