Agency is bugs and uncertainty

Story time! Shortly after Brawl came out, I got pretty good at it. I could beat all my friends without much effort, so I decided to enter in a local tournament. In my first round I went up against the best player in my state, and I managed to hit him once, lightly, over the course of two games. I later became pretty good friends and practiced with him regularly.

At some point I completely eclipsed my non-competitive friends, to the extent that playing with them felt like a chore. All I had to do was put them in certain situations where I knew how they would react and then punish. It became a simple algorithm. Get behind them in shield, wait for the roll, punish. Put them in the air, jump after them, wait for the airdodge, punish. Throw them off the ledge, wait for the jump, punish. It felt like I was playing a CPU.

Meanwhile, I still couldn't reliably beat the best player from my state. One day, after he took off a particularly gruesome stock, I paused and, exasperated, asked for advice. We watched a replay and he showed me how I responded to certain situations in the same way every time, leading to a punish. My habits were less obvious than those of my friends, but they were still habits. He said, "you play like a robot, in a bad way."

So yeah. In that context, I've downgraded friends to CPUs because of their predictability, and been downgraded to a CPU by omega, because of my predictability.

Ancient people, who didn't know any better, saw agency in everything around them. There was a weather god controlling the weather, and a sea god controlling the sea, etc. I wonder to what extent this explains that.

EDIT: This could explain the AI effect too. Once we understand how something works, we don't see it as AI.

If you know yourself well enough to know for sure what you would do in a certain situation, but don't like what you would do, then you consider this mechanistic and not agenty. But you think it is agenty if you like it and think it's a great idea. So this leads to a bias in favor of thinking that whatever you are going to do must be a great idea. You want to think that so that you can think that you are agenty, even if you are not.

I love this post! I've been thinking about it a lot. I think it's mostly wrong, but it's been very stimulating. The main problem is that it wobbles between treating agency in the stated sense of "a capacity to act and make choices", and treating it as intelligence, and treating it as personhood. For one thing, a game player whose moves are predictable will seem dumb, but they still have agency in the stated sense.

Let me see if I can sketch out some slightly improved categories. A predictably moving pendulum is non-agentic, and therefore also not in the class of things that we see as stupid or intelligent: we don't use inverse planning to infer a pendulum's beliefs, preferences, goals, or behavioral intents. When we see an event that's harder to predict or retrodict, like the swaying of a weather vane, the wandering of a tornado, or the identity of a card drawn out from a deck, we reach out for explanations, and planning process explanations may occasionally momentarily be credible, and may be credible more than momentarily if someone is indeed pulling the strings.

As for the relationship between estimation of intelligence and the algorithmic comprehensibility of a goal-directed agent, I propose that AlphaGo seems intelligent even to the people who wrote it while they are playing against it, and that it's rather a judge's uncertainty about individual moves or actions which is a determining factor (alongside the success of the move) in judging intelligence. Or maybe uncertainty about the justification of the move? Either way, a person's ability to examine (or even edit) a mind in detail should not necessarily cause the person to see the mind as being inanimate or non-agentic or non-consequentialist or non-intelligent or non-personlike or unworthy of moral consideration. Whew! I'm glad that the moral status of a thing doesn't depend on an attribution that I can only make in ignorance. That would be devastating!

To recap in more direct terms, predictable things can have agency. Choice-processes are a kind of explanation which may be considered when examining an unpredictable thing, but that doesn't mean that prediction uncertainty always or typically leads to attributions of agency or intelligence (and I really think that a hypothesis space of "agenty or random" is very impoverished).

As for the cognitive antecedents of personification of inanimate things and the conditions that make useful an impersonal, debugging mindset in dealing with people, those are very interesting. From your examples, it seems like there's an implicit belief when we attribute personhood that the thing will respond predictably to displays of anger, and that an impersonal stance is useful or appropriate when a person won't alter their tendency toward a behavior when they hear expressions of negative moral judgement. This would mean that when a judge makes a well-founded attribution of personhood, what they're doing is recognizing that a thing's behavior is well explained by something like "a decision process that considers social norms and optimizes its behavior with an eye toward social consequences". As for what leads to unfounded attributions of personhood, like getting uselessly angry at a beeping smoke alarm, that's still an open question in my mind! Or maybe getting angry at a smoke alarm isn't a kind of personification, and hitting inanimate things that won't stop noising at you is actually a fine response? Hmmmm.

A while ago I asked myself, "If agency is bugs and uncertainty, does that tell us anything about akrasia?" Well, I no longer think that agency is bugs and uncertainty, but re-reading the comments in the thread gave me some new insight into the topic. Oh shit, it's 3:00 am. Maybe I'll sleep on it and write it up tomorrow.

I see your point, but I think you're confusing a partial overlapping with an identity.

There are many bugs/uncertainty that appear as agency, but there are also many bugs/uncertainty which doesn't appear as agency (as you said about true randomness), and there are also behavior that are actually smart and that appear as agency because of smartness (like the way I was delighted with Emacs the first time I realized that if I asked it to replace "blue" with "red", it would replace "Blue" with "Red" and "BLUE" with "RED"), I got the same "feeling of agency" there that I could have on bugs.

So I wouldn't say that agency is bugs, but that we have evolved to mis-attribute attribute agency to things that are dangerous/unpleasant (because it's safest to mis-attribute agency to nothing that doesn't have it, than to not attribute it to something that does have it), the same way our ancestors used to see the sun, storms, volcanoes, ... as having agency.

Agency is something different, hard to exactly pinpoint (philosophers have been going at it for centuries), but that involves ability to have a representation of reality, to plan ahead for a goal, a complexity of representation and ability to explore solution-space in a way that will end up surprising us, not because of bugs, but because of its inherent complexity. And we have been evolved to mis-attribute agency to things which behave in unexpected ways. But that's a bug of our own ability to detect agency, not a feature of agency itself.

To Omega in the Newcomb's problem humans are just automatons without a hint of agency.

I think there's such a thing as too much dissolving of useful concepts. Just because you know how humans run doesn't mean they cease to behave as agents. A human, especially when engaging in goal-directed behavior, is indeed acting as a causal component of the universe that takes in energy and turns it into optimization of events and waste-heat.

My guess is that, in many cases, agency (as in, the capacity to act and make choices) is a manifestation of the observer's inability to explain and predict the agent's actions.

This is my guess too. Agency, intentionality, free will, etc. to me are all cases of the Kolmogorov complexity of the agent being higher than the Kolmogorov complexity of the goal, so we (meaning 'we primates, we humans') tend to assign a mind to the agent, just as we do with our peers.

I think you're probably right about people's feelings regarding buggy software. But for self-reflection, it's different.

In a slogan: agency is self-fulfilling prophecy. What gives us the feeling of agency is the (usually correct) thought that affirming any of a range of thoughts about our own action, would be self-fulfilling. Jenann Ismael -pdf- has a good discussion.

Is this the intentional stance?

So agentiness is having an uncomputable probability distribution?