Comment by Explicator on A Prodigy of Refutation · 2008-09-19T01:23:18.000Z · LW · GW

Ryan--your obervation is true and I agree your resolution...if you don't want to improve, you probably won't. But seeking out related literature for application often speeds up one's rate of progress.

Ian-- Genius demonstrates some convergence...ask the AI a hard math problem, for example, and if it solves it, you know it's smart. On the other hand, if it's smart and doesn't want you to know that, you'll have a hard time finding out anyway. In general, if you know an agent's utility function, you can infer its intelligence based on how well it drives the world towards its target space of preferred outcomes. The uncertainty of knowing the utility function makes this hard. Eli posted on this in more detail very recently.

Tom--This seems useful, though you won't know what's really unsolved versus what's out there on the internet but just not found by you yet. This sounds like a wiki to organize knowledge...further, it's not clear that you should remove problems once you solve them, since you have added some structure to help classify and locate the problem. In general, the internet itself is already functioning as your database, but you could make a subset which prunes itself more efficiently over the problems we care about.

Michael--an AI that "sucks out one's utility function" and doesn't lead to a failure mode itself requires extrapolation of at least one human. Hopefully, many different humans extrapolate similarly...the more this is the case, the less one needs a complicated CEV weighting system. In the extreme case, it could be that 1 human leads to the same outcome as some CEV of all humanity. But this seems risky: if most but not all of humanity extrapolates to one outcome, you increase your chances of getting there by extrapolating more than one person and having them "vote" (assume they are randomly selected, and this follows by basic statistics). There seems to be little value in designing weighting schemes now, since there is more urgent work to be done for the people smart enough to make progress on that problem. So we seem to agree.