AI cooperation is already studied in academia as "program equilibrium"

The illustrations I find most vivid are from the past. Pick some particular technical idea that seems like a "big deal" to you, and then look at its penetration into academic and popular consciousness over time. Diagonalization arguments are a reasonable example. Cantor published over a century ago. The argument is simple enough to explain to an interested 11 year old in less than two hours and repairs basic confusions that many reasonably smart people have about "infinity"... and it remains nearly unknown among non-mathematicians to the present day.

Now imagine the tiny fraction of the population in 1910, 1930, and 1950 who knew about it, and the tricks they could do that "mere mortals" could not. That kind of stuff propagates slowly. Maybe faster now, what with the internet and paywalls slowly coming down and Wikipedia and so on? But still pretty slowly. Kolmogorov complexity has been an available thought for half a century already!

It would not surprise me if every insight needed for AGI had already been published somewhere already, but the separate ideas have just not yet been noticed, collated, synthesized, and reduced to practice. I find this thought sobering.

One of the real tricks is to know what the keywords are, and the best way I know to do that is to participate in academic specialties enough to pick up the "chalk board culture" of a new research group. Grad school (in addition to all the other stuff) is, in some sense, joining a research group and hanging out with them enough to pick up their chalk board culture. Not having access to this is one of the problems that an autodidact trying to make it with book learning and nothing but book learning can run into. Access to the chalk board cultures is partly valuable for helping you see what other smart people think are important things to have read. There's other value as well (like the way a chalk board culture is sometimes a locus of knowledge production), but the advice on keywords and authors is a good chunk of the value.

Actually, I had one just last night. I was starting an anthology on Buddhist philosophy (Empty Words) which I had downloaded from library.nu, and the very first essay was arguing that Nagarjuna's Verses on the Heart of the Middle Way and Sextus Empiricus's Against books espoused a similar skeptical approach to metaphysical commitments (drawing on Kripke & Wittgenstein) - the exact approach I had mused upon at length before, with some of the same passages I would have chosen.

'Huh', I thought, 'I really should have expected someone to have thought that before, but I guess I didn't because I was surprised and angered (since "my" idea had been stolen) to realize where this essay was going.'

Fortunately, I hadn't gotten around to carefully studying and making notes towards an essay, so I didn't lose too much time to my failure to check whether the idea had already been done. (Unlike the time I re-invented longevity insurance, eg.)

Try a bunch of queries; read good-looking papers, not for their results, but their discussion of background information. There's no real good solution to the problem of known unknowns: "I'm sure this has been researched, but what's the exact phrase or name it's been given in academia?" I suspect this is one of the benefits of being in the community - when you discuss something or mention an idea, they can say 'ah yes, X' and now you have not the problem of an unknown known but the much much easier problem of a known unknown which you can just punch into Google Scholar and get to work.

Out of curiosity, via what route did you run across these?

If D is the proof length, surely your reasoning should lead to something like log(T*N) rather than log(T)*N? It seems to me that N people working for T days is T*N person-days, so the two variables should be mostly exchangeable, it's weird to see only one of them under the logarithm...

Since we expect P != NP, finding proofs is an exponential search problem.

P != NP does not imply that one can't do better than an exponential search for finding proofs. For instance an O(n^log(n)) algorithm for finding proofs would not contradict P != NP.

I wonder if EY is partly to blame.

For example, the entire personality psychology research program was built on self- and peer-evaluations, just asking people "are you a pretty confident person?" and so on. That alone is amazing. It's like using the motion of an electron you don't understand to predict the motion of a proton you don't understand.

You say that as if it were an obviously bad thing. But, well, if you don't understand the motion of electrons or protons, but want to figure them out, what else can you do than start conducting easy experiments and start looking for regularities that you could build your theory on?

And yes, many research programs are partially built on self- and peer-evaluations. But those evaluations are also checked for stability between rating occasions and between raters, and then looked at in order to find stable correlations between the evaluation and objective measures, ones which persist after other factors have been controlled for.

Sure, it might be better if you could find some completely objective measures to start out from... but the human brain has great tools to evaluate other human brains. Humans have effectively been handed a ready-made toolkit, parts of which evolved for the express purpose of extracting complex regularities about the behavior of themselves and others and distilling their observations into a highly compressed format. If you devise a measure which achieves good consistency and inter-rater reliability, that's strong evidence of something, and you'd be crazy not to take advantage of it to try to figure out what it means.

Now I'm not saying that psychology would be perfect, or even great. There is a lot of crap, and the theories are still a mess. But I don't think it's as hopelessly bad as you're implying it to be, either.

What do you think about statistics/machine learning? Also what exactly do you mean when you say the situation in "logic" is scarcely better than in psychology? Do you mean mathematical logic/model theory? GOFAI? Logic in analytic philosophy?

One area where psychology journals are said to be leading the way is encouraging a discussion of effect sizes. It's shocking to me that all journals don't do this.

No, I didn't mean to imply that these papers focus on quining.