What Makes A Good Measurement Device?

I think this is going to be wrong as an approach. Weight and temperature are properties of physical systems at specific points in time, and can be measured coherently because we understand laws about those systems. Alignment could be measured as a function of a particular system at a specific point in time, once we have a clear understanding of what? All of human values? 

I'm with Davidmanheim here, it seems this idea could benefit from reading in measurement theory, or at least recognizing a discrepancy that undermines the analogy. I'll get into that a bit, but to start, the post was definitely positive food for thought.

If you're measuring actual temperature, you have some measure options there too, but fundamentally it's a quality of the material under study. If you're measuring "the" perceived temperature, it's an interaction between "the average person" and the material, and sticking fingers in is probably a good measure. Yes, temperature and perceived temperature will correlate, but if the thing you're measuring exists only in someone's head, you're going to have to go to their head for the measurement (also see psychophysics).

"Train[ing] a net to replicate human reports" is not obviously less useful than "actual" scales. Human reports may in fact be the most construct-valid measure. (Though I do agree that leaving these reports in the form of natural language rather than attempted quantifications would indeed be ambiguous, and if we lack face-valid quantitative measures, we will have to develop them from somewhere, probably with those open-ended responses as a foundation.)

Although human reports may be noisy, so are all measures. The thermometer has an implicit +/- margin of error. It seems very precise to us, but human judgments of attributes can also be reliable (in that lots of people agree) and precise (in that the error bars are narrow). For example, if I asked a lot of people to rate the perceived precision of various measures on a scale of 1=extremely noisy to 100=extremely precise, I expect there to be a decent amount of consistency in the rank ordering of those ratings, for thermometers to score highly, and for at least some of the average perceived precisions to flash pleasantly narrow error bars.

But because even the lowest-variance perceptions vary a lot between people (vs. the variability in temperature readings from a thermometer), I do suspect you're not going to get readings that are "approximately-deterministically" useful indicators for lots of perceptual domains, such as alignment. But you'll get indicators that "far-from-deterministically-but-reliably" predict variance in criterion variables. In the end, we're pessimistic and optimistic about the same things; I just don't think it's because human reports are inherently the wrong tool, it's because the attribute of interest is a psychological construct rather than a conveniently-precisely-measurable-physical property. Again, the post was good food for thought - just as measurement of temperature has improved and gotten more precise (touch it -> use mercury -> use radiation), maybe the methods we use for psychological measurement will develop and improve, with hope for alignment.

It seems to me that you fail to understand the natural direction of interpretation. Given an "object" O, an "interpretation" of O in terms of I is a map M: I -> O that preserves some structure from I into O. Not the other way around.

Your physical scale (or your physical thermometer) is before all a physical object S. Calling such a physical object S a scale is asserting the existence of an "interpretation map" M: I -> S that preserves the structure of I into S, in a relevant and satisfying way.  What is the domain I here? For an engineer, it's going to be the theory of Newtonian mechanics, the primitives of which being the concepts of mass, space, time, and force. On top of that, a very explicit notion of weight/gravitational interaction between two objects has to be given. 

That is, the domain I of the interpretation map actually contains more than just the primitive concepts, it also gives an explicit form of F in the F = ma part of Newtonian mechanics. You can for example go with F = -mg if that satisfies you, or go with the less incorrect -GmM/r^2 form. Still, both forms require as data the earth's mass and a knowledge of G (either directly, either indirectly by a process of calibration).

So suppose that you have such an interpretation map M: I -> S into your object. S is now called a scale if it preserves the whole structure from I in a way you judge satisfactory. That is, if it shows the expected numbers predicted on the domain of M, and if it preserves the relative order of mass/weight difference (given two objects O1 and O2 to be measured with S, if the domain I says that the mass of O1 is superior to the mass of O2, then S agrees). Note that if you were rigorous, it is not S that is interpreted to be a scale, but M(I).  It would be a mistake to think that S is uniquely and completely characterized by M. In fact, even M(I) is not completely or uniquely characterized by M (as in, it is possible to find different maps from the same I into M(I), to find different I, etc.).

This is where lies the problem with language. The "uniqueness part" is very very wrong. There exists a lot of different and incompatible domains that are yet mapped satisfyingly into your sentence. A lot of wordplay are actually based on that. Language is also non-associative, but the parenthesizing is never written explicitly. It is also non-commutative (both within a sentence, or between sentences themselves), but that's less of a problem.

What is important to understand in any case, is that this interpretation process is not a canonical or choice-free one. A decision is made at some point, by someone, so it is inherently subjective. Physics being extremely constraining and authoritative, these choices and fundamental ambiguities are generally barely visible in the daily world/engineering world. For language, however, you cannot avoid them. It gets even worse when the very same word of the dictionnary can refer to multiple incompatible concepts already.

When it comes to your post, you're basically asking people to try to find the good domains/theories that explain what happens in a net. First, a net is a black box, so good luck with that. Secondly, people are trying to do so for decades and are failing hard.  Finding the good "I" is just an extremely difficult task.  Even if you had some good "I", asserting that what you see in a black box is indeed well interpreted by such a specific domain (and not another one) won't be convincing (most likely). Any metric people are using to quantify what happens in a net is already playing the role of a scale. It's just not an interesting scale, generally.

I'm generally pretty pessimistic about research-automation proposals analogous to "train a net to mimic a human sticking their finger in some stuff and reporting hotness".

I haven’t read your whole post yet, but I predict that this is because you wouldn’t have reason to believe that you can extrapolate out of the distribution that is measurable by humans.