good in practice, but has conceptual problems in principle

I think that my schemes are pretty purely theoretical; for now I'm happy to not even be in the running for "good in practice" :)

I think it's very reasonable to question whether my abstractions make any sense, and that this is a likely place for my entire approach to break down.

That said, I think there is definitely a superficially plausible meaning of "aligned at capacity c": it's an agent who is "doing its best" to do what we want. If attacking abstractions I think the focus should be on whether (a) that intuitive concept would be good enough to run the argument, and (b) whether the intuitive concept is actually internally coherent / is potentially formalizable.

If we replace "aligned at capacity $c$" with "is trying its best to do what we want," it seems like this objection falls apart.

If agent $N$ is choosing what values agent $N+1$ should maximize, and it picks $r$, and if it's clear to humans that maximizing $r$ is at odds with human interests (as compared to e.g. leaving humans in meaningful control of the situation)---then prima facie agent $N$ has failed to live up to its contract of trying to do what we want.

Now that could certainly happen anyway. For example, agent $N$ could not know how to tell agent $N+1$ to "leave humans in meaningful control of the situation." Or agent $N$ could be subhuman in important respects, or could generalize in an unintended way, etc.

But those aren't decisive objections to the concept of "trying to do what we want" or to the basic analysis plan. Those feel like problems we should consider one by one, and which I have considered and written about at least a little bit. If you are imagining some particular problem in this space it would be good to zoom in on it.

(On top of that objection, "aligned at capacity $c$" was intended to mean "for every input, tries to do what we want, and has competence $c$" not to mean "is aligned for everyday inputs." Whether that can be achieved is another interesting question.)

As an aside: I think that benign is more likely to be a useful concept than "aligned at capacity $c$," though it's still extremely informal.

I'm not sure you've gotten quite ALBA right here, and I think that causes a problem for your objection. Relevant writeups: most recent and original ALBA.

As I understand it, ALBA proposes the following process:

1. H trains A to choose actions that would get the best immediate feedback from H. A is benign (assuming that H could give not-catastrophic immediate feedback for all actions and that the learning process is robust). H defines the feedback, and so A doesn't make decisions that are more effective at anything than H is; A is just faster.
2. A (and possibly H) is (are) used to define a slow process A+ that makes "better" decisions than A or H would. (Better is in quotes because we don't have a definition of better; the best anyone knows how to do right now is look at the amplification process and say "yep, that should do better.") Maybe H uses A as an assistant, maybe a copy of A breaks down a decision problem into parts and hands them off to other copies of A, maybe A makes decisions that guide a much larger cognitive process.
3. The whole loop starts over with A+ used as H.

The claim is that step 2 produces a system that is able to give "better" feedback than the human could -- feedback that considers more consequences more accurately in more complex decision situations, that has spent more effort introspecting, etc. This should make it able to handle circumstances further and further outside human-ordinary, eventually scaling up to extraordinary circumstances. So, while you say that the best case to hope for is $r_i\to r$, it seems like ALBA is claiming to do more.

A second objection is that while you call each $r_i$ a "reward function", each system is only trained to take actions that maximize the very next reward they get (not sum of future rewards). This means that each system is only effective at anything insofar as the feedback function it's maximizing at each step considers the long-term consequences of each action. So, if $r_i\to r$, we don't have reason to think that the system will be competent at anything outside of the "normal circumstances + a few exceptions" you describe -- all of its planning power comes from $r_i$, so we should expect it to be basically incompetent where $r_i$ is incompetent.

FWIW, this also reminded me of some discussion in Paul's post on capability amplification, where Paul asks whether we can even define good behavior in some parts of capability-space, e.g.:

The next step would be to ask: can we sensibly define “good behavior” for policies in the inaccessible part H? I suspect this will help focus our attention on the most philosophically fraught aspects of value alignment.

I'm not sure if that's relevant to your point, but it seemed like you might be interested.