We Humans Learn About Our Values

I'd kinda like to wrap this whole section in a thought-bubble, or quote block, or color, or something, to indicate that the entire section is "what it looks like from inside a human's mind". So e.g. from inside my mind, it looks like we humans learn about our values. And then outside that bubble, we can ask "are there any actual 'values' which we're in fact learning about"?

Indeed, our beliefs-about-values can be integrated into the same system as all our other beliefs, allowing for e.g. ordinary factual evidence to become relevant to beliefs about values in some cases.

Super unclear to the uninitiated what this means. (And therefore threateningly confusing to our future selves.)

Maybe: "Indeed, we can plug 'value' variables into our epistemic models (like, for instance, our models of what brings about reward signals) and update them as a result of non-value-laden facts about the world."

But clearly the reward signal is not itself our values.

Ahhhh

Maybe: "But presumably the reward signal does not plug directly into the action-decision system."?

Or: "But intuitively we do not value reward for its own sake."? 

in a hand-wavy reinforcement-learning-esque sense

language

an agent could aim to pursue any values regardless of what the world outside it looks like; “how the external world is” does not tell us “how the external world should be”.

Extremely delicate wording dancing around the "should be" vs "should be according to me" distinction, with embeddedness allowing facts to update "should be according to me" without crossing the is-ought gap... in principle.

It does seem like humans have some kind of physiological “reward”, in a hand-wavy reinforcement-learning-esque sense, which seems to at least partially drive the subjective valuation of things.

Hrm... If this compresses down to, "Humans are clearly compelled at least in part by what 'feels good'." then I think it's fine. If not, then this is an awkward sentence and we should discuss.

an agent could aim to pursue any values regardless of what the world outside it looks like;

Without knowing what values are, it's unclear that an agent could aim to pursue any of them. The implicit model here is that there is something like a value function in DP which gets passed into the action-decider along with the world model and that drives the agent. But I think we're saying something more general than that.

but the fact that it makes sense to us to talk about our beliefs

Better terminology for the phenomenon of "making sense" in the above way?

a “map” of our values

guess at their own values

Every time the wording of a sentence implies that there are, in fact, some values which someone has or estimates, I picture the adorable not-so-sneaky elephant.

“learn” in the sense that their behavior adapts to their environment.

I want a new word for this. "Learn" vs "Adapt" maybe. Learn means updating of symbolic references (maps) while Adapt means something like responding to stimuli in a systematic way.

know our own values

... there's a whole fucking elephant swept under that rug. I can see its trunk peaking out. It's adorable how sneaky that elephant thinks it's being.

The internal heuristics or behaviors “learned” by an adaptive system are not necessarily “about” any particular external thing, and don’t necessarily represent any particular external thing

I give up.

Adaptive systems “learn” things, but they don’t necessarily “learn about” things; they don’t necessarily have an internal map of the external territory.

So much screaming

symbolic representation of the environment, and they update those symbols over time to (hopefully) better match the environment

more scream

behavior adapts to their environment

scream