LessWrong 2.0 Reader

Re the OpenAI o-series and search, my initial prediction is that Q*/MCTS search will work well on problems that are easy to verify and and easy to get training data for, and not work if either of these 2 conditions are violated, and secondarily will be reliant on the model having good error correction capabilities to use the search effectively, which is why I expect we can make RL capable of superhuman performance on mathematics/programming with some rather moderate schlep/drudge work, and I also expect cost reductions such that it can actually be practical, but I'm only giving a 50/50 chance by 2028 for superhuman performance as measured by benchmarks in these domains.

I think my main difference from you, Thane Ruthenis is I expect costs to reduce surprisingly rapidly, though this is admittedly untested.

This will accelerate AI progress, but not immediately cause an AI explosion, though in the more extreme paces this could create something like a scenario where programming companies are founded by a few people smartly managing a lot of programming AIs, and programming/mathematics experiencing something like what happened to the news industry from the rise of the internet, where there was a lot of bankruptcy of the middle end, the top end won big, and most people are in the bottom end.

Also, correct point on how a lot of people's conceptions of search are babble-and-prune, not top down search like MCTS/Q*/BFS/DFS/A* (not specifically targeted at sunwillrisee

By contrast, my understanding is that the sort of search John is talking about retargeting isn't the brute-force babble-and-prune algorithms, but a top-down heuristical-constraint-based search [LW · GW].
 

Maybe someone else could moderate it?

There's no good candidate for a simple, legible, easily-obtained, and agreeable-to-most metric.  Before-and-after polling of patients is probably closest we can get.

That said, the dimensions of quality that the FDA concerns itself with (including physical functioning, self-reported pain, and other easily- and not-easily-measured things) is likely close enough to "improves quality of life" that it's not necessary to have a new direction.  

Perhaps you could identify some drugs that you think would improve quality of life, and work backwards to the metrics that prove to you that they do so.

this concern sounds like someone walking down a straight road and then closing their eyes cause they know where they want to go anyway

This doesn't sound like a good analogy at all. A better analogy might be a stylized subway map compared to a geographically accurate one. Sometimes removing detail can make it easier to process.

A policeman sees a drunk man searching for something under a streetlight and asks what the drunk has lost. He says he lost his keys and they both look under the streetlight together. After a few minutes the policeman asks if he is sure he lost them here, and the drunk replies, no, and that he lost them in the park. The policeman asks why he is searching here, and the drunk replies, "this is where the light is".

I've always been sympathetic to the drunk in this story.  If the key is in the light, there is a chance of finding it.  If it is in the dark, he's not going to find it anyway so there isn't much point in looking there.

Given the current state of alignment research, I think it's fair to say that we don't know where the answer will come from.  I support The Plan [LW · GW] and I hope research continues on it.  But if I had to guess, alignment will not be solved via getting a bunch of physicists thinking about agent foundations.  It will be solved by someone who doesn't know better making a discovery they "wasn't supposed to work".

On an interesting side here a fun story about experts repeatedly failing to make an obvious-in-hindsight discovery because they "knew better".

Seems to me that there is always some friction, some lack of information, etc., so "this requires market failure by definition" basically means "this happens in the real world".

Agreed. Simply focusing on physics post-docs feels too narrow to me.

Then again, just as John has a particular idea of what good alignment research looks like, I have my own idea: I would lean towards recruiting folk with both a technical and a philosophical background. It's possible that my own idea is just as narrow.

You can buy bulk tamiflu powder here: https://www.selleckchem.com/products/oseltamivir-phosphate-Tamiflu.html And instructions for preparation are here: https://dph.illinois.gov/content/dam/soi/en/web/idph/files/publications/tami-flu-flyer-050316.pdf $300 for 13 doses, which would last you about 6 days, dosing the recommended twice daily during a pandemic. Shelf life of the powder is 2 years. Would any medicine nerd sanity check that I am not missing something essential?

Let's say a company demands that a worker buy a suit worth $10, and gains $1 from it. Then the worker could offer to work for $1.05 less, but without the suit, and that would be more profitable for both the company and the worker.

And the company could simply say no, knowing that the worker has more to lose, relatively, and therefore will be likely to give up and accept the original deal.

Seems to me that at least a part of the intuition behind "exploitation" is that the person with greater negotiation power can precommit to reject even the win/win deals if they are not unbalanced enough in their favor.

To use the metaphor of a growing pie, imagine that there is a button that will magically summon a pie for both of us to share, but only if we both press the button simultaneously. Problem is, you are starving but I am not. So I say that unless you give me 90% of the pie, I refuse to press the button. I will lose some good pie, but I can live with that, and you can't.

Furthermore, this is an iterated game. If you accept to take 10% of the pie and let me take 90%, what happens when we find a similar button tomorrow? Yeah, you will be starving again, and I will be not.

(And this can get even more unfair, when the stronger party can use their advantage to lobby for making the environment even worse for the weaker party. Not sure what would be the proper metaphor here. Making it illegal to eat things other than pies? Making it illegal for two people to press the magical button unless one of them is me?)

Definitely.

Lack of publicly reporting null results was a subtle but huge problem in cognitive neuroscience. It took a while to figure out just how much effort was being wasted running studies that others had already tried and not reported because results were null.

Alignment doesn't have the same journal gatekeeping system that filters out null results, but there's probably a pretty strong tendency to report less on lack of progress than actual progress.

So post about it if you worked hard at something and got nowhere. This is valuable information when others choose their problems and approaches.

I do see people doing this; it would probably be valuable if we did it more.