Concrete vs Contextual values

Both of your attempted counterexamples in chess and wealth have the same structure: it is relative chess-playing ability and relative wealth that fails to go FOOM, because they are directly opposed by other FOOMs on the same curve; it is obvious that relative ability cannot go FOOM for all players in a game. Economic growth is still an exponential curve and so is the chess-playing efficiency of computers compared to stationary targets (that is, us).

The question of relative-ability FOOM versus global-shared FOOM was precisely the substance of the dispute between myself and Robin.

I would also suggest that the distribution of wealthy humans is more fat-tailed than the distribution of computer chessplayer efficiences precisely because of the increased self-interaction of human wealth.

I think the intuition you have for recursive self-improvement is that of a machine running essentially the same program at each stage, just faster. That's not what's meant. Human brains aren't sped-up chimp brains; they have some innate cognitive modules that allow them to learn, communicate, model cause and effect, and change the world in ways chimps simply can't. We don't know what cognitive modules are even possible, let alone useful; but it seems clear there's 'plenty of room at the top', and that a recursively self-improving intelligence could program and then use such modules. Not to go Vernor Vinge on you, but if a few patched-together modules made the difference between chimp and human technology, then a few deliberately engineered ones could leap to what we consider absurd levels of control over the physical world.

Presumably, contextual effects could also be positive rather than negative, and could therefore make FOOMs more likely rather than less? To adapt the game analogy slightly, it depends whether your opponents are improving faster than your team mates (and whether your improving tends to attract or repel better team mates).

Physics probably isn't becoming a tougher opponent over time; society probably is. Part of the trick may lie in working with it rather than against it.

What is Bill Gates, but wealth gone FOOM?

What makes the intelligence cycle zero-sum? What devalues the 10 MIPs advance? After all, the goal is not to earn a living with the prize money brought in by an Incredible Digital Turk, but to design superior probability-space searching programming algorithms, using chess as a particular challenge, then to use that to solve other problems which are not moving targets, like machine vision or materials analysis or...alright, I admit to ignorance here. I just suspect that not all goals for intelligence involve competing with/modeling other growing intelligences.

Technological advances (which seem similar enough to "increases in the ability to achieve goals in the world" to be worthy of a tentative analogy) may help some (the 20 MIPs crowd) disproportionately, but don't they frequently still help everyone who implements them? If people in Africa get cellphones, but people in Europe get supercomputers, all people are still getting an economic advantage relative to their previous selves; they can use resources better than they could previously.

Also, if point 3'' is phrased equally as vaguely as 3' (perhaps: "Wealthy people are able to do things to increase the values in 2''."), then it seems much more reasonable. Wealth can be used to obtain information and contacts that giver greater relative wealth-growing advantage, such as "Don't just put it all in the bank," or "My cousin's company is about to announce higher-than-expected earnings," or even "Global hyperinflation is coming, transfer assets to precious metals." Conversely (I think), if point 3' had a formulation sufficiently specific to be similarly limited ("Computers can keep having more RAM installed and thus will have more intelligence over time."), I don't see how that would be an indictment of the general case. What am I missing?

What are you claiming?

I think your statements imply that humans aren't any more intelligent than chimpanzees, because a human isn't able to rise any higher in human society than a chimpanzee is able to rise in chimp society.

If you're making a claim about the utility of increasing intelligence, and saying that we'd be wise to forgo it, then you need to address the game-theoretic / evolutionary force in favor of it.

a system with more compute power will be more capable than one with less (assuming they can get the requisite power supply). Similarly systems that have algorithms with better run time complexities will be more capable.

Is sheer computing power really the issue here? General intelligence isn't going to spontaneously emerge just because you built a really big supercomputer running a really efficient word processor.

I'd argue that intelligence has a contextual nature as well. A simple example would be a computer chess tournament with a fixed algorithm that used as much resources as you threw at it. Say you manage to increase the resources for your team steadily by 10 MIPs per year, you will not win more chess games if another team is expanding their capabilities by 20 MIPs per year.

If you're comparing a randomly selected intelligent system against another randomly selected intelligent system drawn from the same pool, then of course the relative difference isn't going to change as you crank up the general level of intelligence.

But if you compare one of these against anything else as you crank up the general level of intelligence then it's a whole other story. And these other comparisions are pretty much what's at stake here.

You seem to be arguing that one FOOM in the middle of another bunch of faster FOOMs won't be so impressive.

Er, OK, fair enough, and so?

In your wealth example, the contextual constraint affects the feedback process (i.e. capability to self-improve) rather than capability to affect the world. This seems crucial, but it's not clear that contextual constraints on intelligence function in this way.